Chronicle is a cool name right? It’s like science fiction movie’s AI. Chronicle find me the Uranium!
But unfortunately it is just a queue, it is a good one too.
In our project (It was a network project, we got datas from devices and analyzed them) there was a need that 10 million device objects have to be shuffled and send it to get datas from devices. They have to be shuffled because you shouldn’t go more than once to network devices at the same time, it cause too much network activity and it effects network quality in devices.
It is easy right just get the objects and shuffle them.
Collections.shuffle(list);
Nope. Your heap size gonna off the roof by doing this. 10 million object is really huge and object has 30–35 property.
If I remember correctly first time we try this, heap size was over 10GB and application was crashed.
So 10 million data, how can I shuffle them and not causing huge increase on heap size.
I started to research. I wanted to store them in a file and get them randomly, it would be quick and heap free. But how can you do that? Okay you stored your all object to a file line by line but how can you take them by random? File doesn’t have a line number to get data like lines.get(1) and I think I tried to find going to a random line but I couldn’t find anything.
I didn’t wanted to use db. Mongo_(we were using that) _was already doing too much work and it would be too slow to work with and getting datas by random is still a problem.
#java #software-development #queue #solutions #software-engineering
