One problem is security - users may have restricted access not only to tables and fields, but also to individual records (through record-level security).
You also won’t be able to use any business logic defined in AX. It includes both system functionality (virtual company accounts, time-zone handling etc.) and custom code (e.g. display methods).
If you’re developing something for a single customer, you might be able to review what functionality they use, reimplement what’s needed, let them to configure security setup for the user on DB-level etc. If you’re trying to build a more general solution, you’ll have to take into account all functionality that anybody may want to use.
In short, it’s possible, but not recommended. What’s recommended are web services going through AOS, which deals with all AX specifics.