You could use a tool like newrelic to get an idea of where the bottleneck is. But there are so many things that can slow down an app, you’ll probably need some technical help to diagnose and resolve the issue.

go AWS

trust me bro… why trust me?

microsoft = feasible TECHNICALLY, but the worst customer support in the world.

aws is just as scalable and has amazing customer support. it’s easier to understand

digital ocean… this is good for what? they’re a little cheaper, but that’s because they’re… cheaper. you’re obviously not satisfied with him

you could also check out google cloud, they’re probably really good but i don’t know about them much

5 seconds?

For me it looks like that you query to your database each item (Ping pong between application server and database server, the latency between them sums up by the number of items in the database). Some junior developers use ORM layers still they find out they have such a problem. Just search n+1 problem, maybe you can fix it.

this is not infrastructure related (or you use that much less memory and force your system to swap).

hope this helps.

Speaking as a techie having dealt with clients with that problem: You need a techie to help you.

Your whole concept could be flawed to the point where you will find it prohibitively expensive to create the user experience that you’re going for, or there might be a switch to flip.

And the reason that I’m saying that you need help from someone to figure this out is that if you had the skills to find the underlying cause you wouldn’t ask what you just asked; your question would be a lot more specific, and it would be in a more relevant tech subreddit, or you’d right now be reading relevant posts on stackoverflow.

You really need to dig deeper into what’s causing that delay, and you need to try things like emulating 100+ concurrent users to see what happens with that delay. And you need that not only to figure out how to minimize the delay, but also to calculate what resources (CPU, GPU, RAM, network, and so on) will costs you how much on different platforms; so that you can get your numbers right to figure out if your business plan is realistic or not. Can you afford to operate your business? Is your target market willing to pay what you must charge? How much capital do you need to invest in your own hardware (or perhaps operate at a loss on a cloud provider to get early scaleability)? Do you need to rework your software to take advantage of doing some calculations on off-peak hours?

As I know so far, LLMs are pretty GPU-hungry stuff. So I’d go with a dedicated GPU server for the best results, but they cost kinda too much. Anyway I think you should check them out, maybe there would be some affordable options for your team.

As u/victrolla said you need to do some profiling, but I suspect it’s lack of indexes in your SQL database tables.

Needs more profiling. I doubt the API call is the bill of processing time. I’m completely blind about your app architecture, but I’m wondering if you’ve looked into something like google vertex. You’d benefit from hardware acceleration and horizontal scaling of your model. The API component is best to run in something like cloudrun.

Usually what I see for folks like you is they’ve got a python webapp like flask that’s running the model directly. This is a really bad plan. You need to decompose. Push the model into vertex and make an api call to it on the backend.

There’s a lot of possible optimizations but there’s not enough detail to know specifics.

Hi, this is very interesting. I’d like to talk to you more about this since I’m building something in this space. Would love to help you as much as I can in exchange for more details about your usecase and issues. Can we talk in DMs?

I’m in a similar position you are in now. Developing a mobile app that utilizes a couple of computer vision models, using the aws ecosystem for prototyping and finding that server-side development has a pretty big learning curve.