How long should it take to generate Data Table with 50000 rows and 100 column.
How long should it take to generate Data Table with 50000 rows and 100 column.
nya13
Posts: 21Questions: 6Answers: 0
Just running some benchmarks to see if the speed I get is normal due to the large amount of data or something in my code is not correct. So I decided to generate a Data Table with no other processing (render callbacks or cell/row elements processing) involved.
I have used an custom method to generate a data array with **50000 **rows and **100 **columns. It would look something like:
const DATA_ARRAY = [
{ "Column 1": 9999, "Column 2": 9999, "Column 3": 9999, ....... , "Column 100": 9999 },
{ "Column 1": 9999, "Column 2": 9999, "Column 3": 9999, ....... , "Column 100": 9999 },
];
I have a custom method that would auto-generate the "columns" data needed and assigned is to a COLUMNS_DATA constant.
I have added my plain table element <table></table> to the DOM. Since I am using jQuery, I have the jQuery object for table in a $TABLE constant.
I have created a data/object that will contain the "data" and "columns" information:
const SETTINGS_DATA = { 'data': DATA_ARRAY, 'columns': COLUMNS_DATA };
I then ran the Data Tables method to generate the Data Table:
$TABLE.DataTable(SETTINGS_DATA );
From that last call, it took 50 seconds before the table showed up on my page in the Chrome browser. The browser sometimes did the "Aw, Snap" because it ran out of memory.
I did a few more test:
- With 50 columns, it took 23 seconds.
- With 25 columns, it took 9 seconds.
- With 15 columns, it took 5 seconds.
- With 5 columns, it took 2 seconds.
Maybe it would be better to use the server-side option, but trying to see if the speed I am getting is expected for that amount of data.
Answers
It depends on a whole lot of factors! It sounds like you have narrowed it down from server and network to specifically the DataTables initialisation though. 50 seconds seems like a very long time to be for that amount of data.
Can you link to your page so I can profile it and find out what is going on please?
Allan
Hi Allan,
I have the same problem. I can provide links to two instances of DataTables showing a sharp contrast in rendering speed using the same data. One is running DataTables 2.0.3 and the other DataTables 1.13.4. I need to add a firewall rule to allow you to access the pages, however. Can you provide an IP address or CIDR range that I can let through? Perhaps you can send me a direct message. I also need to provide you with a username and password. Let me know. Thanks!
Andrei
Hi Andrei,
Many thanks. I've just sent you a PM with my IP address.
Allan
UDPATE - Here is the response I received from Allan:
Looking at the performance trace, it looks like a lot of the time is spent in the renderers, and I think that is because it is doing DOM manipulation. Messing with the DOM is always very slow, so if it is possible to cut out the DOM stuff, that will help massively (as an experiment, you could have renderCell just return the value without checking classes or anything else.
Also, renderCell is running and doing DOM manipulation for every call to a rendering function - whereas, if it is necessary, it should only be doing it when the type parameter is 'display'. Doing the DOM manipulation when getting sort data for example is unnecessary and will slow things down.
I think with optimisation of the rendering functions, the DataTable performance can be brought a bit more under control.
The rendering function is used more regularly in DT2, which is, I think, why there is such a noticeable performance impact for your application.The renderer is called, not a lot more often, but enough that when you do DOM manipulation in it, it is going to be noticeable. DataTables still caches data, but how it handles the filtering, sorting and type detection has been updated a bit, and I think that is what is causing the issue to be highlighted here.
The quick "win" will be to check for the display data type request - but more generally, I'd suggest moving any DOM manipulation to columns.createdCell, createdRow, or rowCallback (the latter if the DOM needs to update dynamically after initialisation as the other two are only ever called once).
UPDATE - Here are my initial findings:
When I added a check for type == 'display' in the renderers I noticed the following:
In DT1 when type == 'display' the cell currently being rendered has a pre-defined node() of '<td><td>'. Also, all previously rendered cells in the current row have their nodes accessible through the node() API.
In DT2, however, when type == 'display' the nodes of the current row are not available until after the entire row is rendered. In particular table.cell(meta.row, meta.col).node() is undefined, making it difficult to do access the DOM for the cell being rendered.
When type == 'type' the DOM node() is defined and available but when this is the case the renderer functions are called outside of the draw sequence (after the drawCallback and the draw event) resulting in the UI "flashing" if any DOM manipulation is performed.