Openlr Performance Issue

[sudhaharv77]

Hi,

We are developing the traffic project, we are getting the traffic data from
tomtom(https://traffic.tomtom.com/tsq/hdf/ARE-HDF-OPENLR/149d7921-d010-43cd-8e0d-83e5df700500/content.proto) as of now URL is not open for us. we are using the sample proto data.

Here the issue is, when we try to decode the data using openlr(openlr.decoder.OpenLRDecoder;) its consuming more time to return response( example 1 sec for 1 record).

We have almost 43000 records needs to process approx. its takes almost 40 mins which run by 200 threads.

Could you please help to sort out issue.

Java Sample:

ByteArray openLRByteArray = new ByteArray((byte[]) trafficFlow.getLocation().getOpenlr().toByteArray());
LocationReference locRef = new LocationReferenceBinaryImpl("", openLRByteArray);
Location decodedLoc = new OpenLRDecoder().decode(decParam, locRef);

Thanks,

[ghost]

I have a few questions about the issue:

• What map implementation are you using to decode the records?
• Are you using the maploader-tt-sqlite module and an associated SQLite database?
• If so, how large is the map? Can you perform a count of the records in the Line and Node tables?

The map implementation in the maploader-tt-sqlite was designed as a simple test map format for getting started. For your task I would suggest implementing the MapDatabase interface using a proper spatial database.

[sudhaharv77]

Hi,
Thanks for your valuable response, Please spare some time to read my comments

fyi info we are below dependencies

Please find my answers,

What map implementation are you using to decode the records?

• We are getting the MapData list based on Zone from our DB(Data is coming from GEO db)

- [ ] openlr.map.MapDatabase ojbect is created as below

- [ ] prepare openlr decode param using below

- [ ] finally decoding as below.

finall results is

Are you using the maploader-tt-sqlite module and an associated SQLite database?

Yes

openlr-mapLoader
tt-sqlite
1.3.0

If so, how large is the map? Can you perform a count of the records in the Line and Node tables?

[ghost]

Thanks for posting the info! Could you try reducing the number of threads from 200 down to 10? I think the number of threads is too high and they are likely competing for resources. This might increase the throughput.

You seem to be getting throughput of about (40m * 60s * 1000ms / 43000 records) = 55 milliseconds per record. The OpenLR sqlite database was not really designed for high-performance but rather as a simple example implementation of a map to get started with OpenLR. I think you are just hitting on the limits of sqlite. You could try, for example, implementing MapDatabase using an in-memory copy of the map. This would perform much better.

[sudhaharv77]

Hello Stephen,

Thanks for you valuable feed back.
Yes as you said, we can achieve our goal if sqlite file minimal. As per our records(43000 min) its takes long time to process with 32gig machine. We need to implement In memory concept as you suggest.

i could not found any API's support for IN MEMORY with openlr.

Could you please help us to provide any snippet or steps to implement IN MEMORY.

Thanks.

[ghost]

You are right. We don't have a in-memory implementation of the map database. This is something we would like to do in the future. To achieve this:

• the following classes would need to be implemented openlr.map.MapDatabase, openlr.map.Line and openlr.map.Node.
• All lines and nodes of the map would be loaded into memory. Spatial indexes would need to be built for all lines and all nodes to enable quick searching around a point. An RTree implementation such as https://locationtech.github.io/jts/javadoc/org/locationtech/jts/index/strtree/STRtree.html might be good for this task.
• Indexes would also need to be built to enable quick retrieval of line and node by ID.
• Some geometrical operations would need to be implemented such as getGeoCoordinateAlongLine. A library such as JTS could also be used here to reduce the implementation work.

[sudhaharv77]

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation
is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards
Sudhahar

[duel-adn]

Hello @sudhaharv77,
we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me.

Best,
antonio

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation
is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards
Sudhahar

[sudhaharv77]

Hi Antonio Yes I am interested, could please share more details Regards Sudhahar

…

On Thu, Mar 25, 2021, 5:14 PM antonio di noto ***@***.***> wrote: Hello @sudhaharv77 <https://github.com/sudhaharv77>, we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me. Best, antonio Hi stephencurran, Thanks again, Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future? Can you suggest instead Java, is there any other technologies to process the data faster with Openlr. Regards Sudhahar — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q> .

[duel-adn]

Hi Sudhahar, We achieve these performances because we've developed our own graph database specifically for this task and connected it to the OpenLR implementation on github. We're using it with TomTom's Multinet of a large portion of the world (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per second (it depends on the density of the graph in the area). If you are interested, we could offer a simple REST APIs for decoding OpenLR in batches. Can you share some information about your project? best, antonio On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 ***@***.***> wrote:

…

Hi Antonio Yes I am interested, could please share more details Regards Sudhahar On Thu, Mar 25, 2021, 5:14 PM antonio di noto ***@***.***> wrote: > Hello @sudhaharv77 <https://github.com/sudhaharv77>, > we have a very fast OpenLR decoder implementation (6k-50k identifiers per > second on a 4 core machine with 16GB of RAM). If you are still interested, > you can contact me. > > Best, > antonio > > Hi stephencurran, > > Thanks again, > > Sorry for the late reply, i understand that openlr do not have in-memory > implementation > is it any road map to implement in-memory in future? > > Can you suggest instead Java, is there any other technologies to process > the data faster with Openlr. > > Regards > Sudhahar > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #33 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q> .

-- Antonio Di Noto, CTO Duel S.p.A. via del Poggio Laurentino, 66 - 00144 Roma Tel. +39 06.5195951 Linkedin http://it.linkedin.com/in/antoniodinoto/ http://www.dueltv.com Please consider the environment before printing this e-mail

[sudhaharv77]

Hi Antonio, Let me explain our project, We are doing Advanced Traffic Management System(ATMS) for GULF country, we already got the API to get live traffic data from TOMTOM ( https://traffic.tomtom.com/tsq/hdf-detailed/ARE-HDF_DETAILED-OPENLR/xxxxxxxxx ) We are calling every 2 mins to get updated traffic speed data from the above URL to update our GIS map TOMTOM providing 55k records as encrypted format then we are decoding each time using OPENLR ( https://github.com/tomtom-international/openlr/tree/master/decoder) We are unable to complete the whole process with in 2 min, I need help from you to compete the whole process with in 2 min, Or any other method to do i have used below snippet to process the tomtom inputstream object [image: image.png] Regards Sudhahar On Thu, Mar 25, 2021 at 6:36 PM antonio di noto ***@***.***> wrote:

…

Hi Sudhahar, We achieve these performances because we've developed our own graph database specifically for this task and connected it to the OpenLR implementation on github. We're using it with TomTom's Multinet of a large portion of the world (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per second (it depends on the density of the graph in the area). If you are interested, we could offer a simple REST APIs for decoding OpenLR in batches. Can you share some information about your project? best, antonio On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 ***@***.***> wrote: > Hi Antonio > > Yes I am interested, could please share more details > > Regards > Sudhahar > > > > On Thu, Mar 25, 2021, 5:14 PM antonio di noto ***@***.***> > wrote: > > > Hello @sudhaharv77 <https://github.com/sudhaharv77>, > > we have a very fast OpenLR decoder implementation (6k-50k identifiers per > > second on a 4 core machine with 16GB of RAM). If you are still > interested, > > you can contact me. > > > > Best, > > antonio > > > > Hi stephencurran, > > > > Thanks again, > > > > Sorry for the late reply, i understand that openlr do not have in-memory > > implementation > > is it any road map to implement in-memory in future? > > > > Can you suggest instead Java, is there any other technologies to process > > the data faster with Openlr. > > > > Regards > > Sudhahar > > > > — > > You are receiving this because you were mentioned. > > Reply to this email directly, view it on GitHub > > < > #33 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q > > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #33 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q > > . > -- Antonio Di Noto, CTO Duel S.p.A. via del Poggio Laurentino, 66 - 00144 Roma Tel. +39 06.5195951 Linkedin http://it.linkedin.com/in/antoniodinoto/ http://www.dueltv.com Please consider the environment before printing this e-mail — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZBLFGWQ6U4CZUNFTMJVBTTFMG2VANCNFSM4JVW3L7Q> .

[sudhaharv77]

Hi Antonio, Additional information for the decode below, [image: image.png] [image: image.png] Regards Sudhahar On Tue, Mar 30, 2021 at 2:55 PM Sudhahar Vaithilingam < ***@***.***> wrote:

…

Hi Antonio, Let me explain our project, We are doing Advanced Traffic Management System(ATMS) for GULF country, we already got the API to get live traffic data from TOMTOM ( https://traffic.tomtom.com/tsq/hdf-detailed/ARE-HDF_DETAILED-OPENLR/xxxxxxxxx ) We are calling every 2 mins to get updated traffic speed data from the above URL to update our GIS map TOMTOM providing 55k records as encrypted format then we are decoding each time using OPENLR ( https://github.com/tomtom-international/openlr/tree/master/decoder) We are unable to complete the whole process with in 2 min, I need help from you to compete the whole process with in 2 min, Or any other method to do i have used below snippet to process the tomtom inputstream object [image: image.png] Regards Sudhahar On Thu, Mar 25, 2021 at 6:36 PM antonio di noto ***@***.***> wrote: > Hi Sudhahar, > > We achieve these performances because we've developed our own graph > database specifically for this task and connected it to the OpenLR > implementation on github. > We're using it with TomTom's Multinet of a large portion of the world > (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per > second (it depends on the density of the graph in the area). > > If you are interested, we could offer a simple REST APIs for decoding > OpenLR in batches. > > Can you share some information about your project? > > best, > > antonio > > On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 ***@***.***> > wrote: > > > Hi Antonio > > > > Yes I am interested, could please share more details > > > > Regards > > Sudhahar > > > > > > > > On Thu, Mar 25, 2021, 5:14 PM antonio di noto ***@***.***> > > wrote: > > > > > Hello @sudhaharv77 <https://github.com/sudhaharv77>, > > > we have a very fast OpenLR decoder implementation (6k-50k identifiers > per > > > second on a 4 core machine with 16GB of RAM). If you are still > > interested, > > > you can contact me. > > > > > > Best, > > > antonio > > > > > > Hi stephencurran, > > > > > > Thanks again, > > > > > > Sorry for the late reply, i understand that openlr do not have > in-memory > > > implementation > > > is it any road map to implement in-memory in future? > > > > > > Can you suggest instead Java, is there any other technologies to > process > > > the data faster with Openlr. > > > > > > Regards > > > Sudhahar > > > > > > — > > > You are receiving this because you were mentioned. > > > Reply to this email directly, view it on GitHub > > > < > > > #33 (comment) > > >, > > > or unsubscribe > > > < > > > https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q > > > > > > . > > > > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < > #33 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q > > > > . > > > > > -- > Antonio Di Noto, CTO > Duel S.p.A. > via del Poggio Laurentino, 66 - 00144 Roma > Tel. +39 06.5195951 > Linkedin http://it.linkedin.com/in/antoniodinoto/ > http://www.dueltv.com > Please consider the environment before printing this e-mail > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#33 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHZBLFGWQ6U4CZUNFTMJVBTTFMG2VANCNFSM4JVW3L7Q> > . >

[sudhaharv77]

Hi Could you please help to update on my Queries Regards Sudhahar On Tue, Mar 30, 2021, 4:03 PM Sudhahar Vaithilingam < ***@***.***> wrote:

…

Hi Antonio, Additional information for the decode below, [image: image.png] [image: image.png] Regards Sudhahar On Tue, Mar 30, 2021 at 2:55 PM Sudhahar Vaithilingam < ***@***.***> wrote: > Hi Antonio, > Let me explain our project, > We are doing Advanced Traffic Management System(ATMS) for GULF country, > we already got the API to get live traffic data from TOMTOM ( > https://traffic.tomtom.com/tsq/hdf-detailed/ARE-HDF_DETAILED-OPENLR/xxxxxxxxx > ) > > We are calling every 2 mins to get updated traffic speed data from the > above URL to update our GIS map > > TOMTOM providing 55k records as encrypted format then we are > decoding each time using OPENLR ( > https://github.com/tomtom-international/openlr/tree/master/decoder) > We are unable to complete the whole process with in 2 min, > > I need help from you to compete the whole process with in 2 min, Or any > other method to do > > i have used below snippet to process the tomtom inputstream object > > [image: image.png] > > Regards > Sudhahar > > > > > > > > > On Thu, Mar 25, 2021 at 6:36 PM antonio di noto ***@***.***> > wrote: > >> Hi Sudhahar, >> >> We achieve these performances because we've developed our own graph >> database specifically for this task and connected it to the OpenLR >> implementation on github. >> We're using it with TomTom's Multinet of a large portion of the world >> (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per >> second (it depends on the density of the graph in the area). >> >> If you are interested, we could offer a simple REST APIs for decoding >> OpenLR in batches. >> >> Can you share some information about your project? >> >> best, >> >> antonio >> >> On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 ***@***.***> >> wrote: >> >> > Hi Antonio >> > >> > Yes I am interested, could please share more details >> > >> > Regards >> > Sudhahar >> > >> > >> > >> > On Thu, Mar 25, 2021, 5:14 PM antonio di noto ***@***.***> >> > wrote: >> > >> > > Hello @sudhaharv77 <https://github.com/sudhaharv77>, >> > > we have a very fast OpenLR decoder implementation (6k-50k >> identifiers per >> > > second on a 4 core machine with 16GB of RAM). If you are still >> > interested, >> > > you can contact me. >> > > >> > > Best, >> > > antonio >> > > >> > > Hi stephencurran, >> > > >> > > Thanks again, >> > > >> > > Sorry for the late reply, i understand that openlr do not have >> in-memory >> > > implementation >> > > is it any road map to implement in-memory in future? >> > > >> > > Can you suggest instead Java, is there any other technologies to >> process >> > > the data faster with Openlr. >> > > >> > > Regards >> > > Sudhahar >> > > >> > > — >> > > You are receiving this because you were mentioned. >> > > Reply to this email directly, view it on GitHub >> > > < >> > >> #33 (comment) >> > >, >> > > or unsubscribe >> > > < >> > >> https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q >> > > >> > > . >> > > >> > >> > — >> > You are receiving this because you commented. >> > Reply to this email directly, view it on GitHub >> > < >> #33 (comment) >> >, >> > or unsubscribe >> > < >> https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q >> > >> > . >> > >> >> >> -- >> Antonio Di Noto, CTO >> Duel S.p.A. >> via del Poggio Laurentino, 66 - 00144 Roma >> Tel. +39 06.5195951 >> Linkedin http://it.linkedin.com/in/antoniodinoto/ >> http://www.dueltv.com >> Please consider the environment before printing this e-mail >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#33 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AHZBLFGWQ6U4CZUNFTMJVBTTFMG2VANCNFSM4JVW3L7Q> >> . >> >

[duel-adn]

Hello, we better continue this discussion in private, my e-mail is: a.dinoto at dueltv dot com, can you send me yours? On Mon, Apr 26, 2021 at 1:21 PM sudhaharv77 ***@***.***> wrote:

…

Hi Could you please help to update on my Queries Regards Sudhahar On Tue, Mar 30, 2021, 4:03 PM Sudhahar Vaithilingam < ***@***.***> wrote: > Hi Antonio, > > Additional information for the decode below, > > [image: image.png] > > > [image: image.png] > > Regards > Sudhahar > > On Tue, Mar 30, 2021 at 2:55 PM Sudhahar Vaithilingam < > ***@***.***> wrote: > >> Hi Antonio, >> Let me explain our project, >> We are doing Advanced Traffic Management System(ATMS) for GULF country, >> we already got the API to get live traffic data from TOMTOM ( >> https://traffic.tomtom.com/tsq/hdf-detailed/ARE-HDF_DETAILED-OPENLR/xxxxxxxxx >> ) >> >> We are calling every 2 mins to get updated traffic speed data from the >> above URL to update our GIS map >> >> TOMTOM providing 55k records as encrypted format then we are >> decoding each time using OPENLR ( >> https://github.com/tomtom-international/openlr/tree/master/decoder) >> We are unable to complete the whole process with in 2 min, >> >> I need help from you to compete the whole process with in 2 min, Or any >> other method to do >> >> i have used below snippet to process the tomtom inputstream object >> >> [image: image.png] >> >> Regards >> Sudhahar >> >> >> >> >> >> >> >> >> On Thu, Mar 25, 2021 at 6:36 PM antonio di noto ***@***.***> >> wrote: >> >>> Hi Sudhahar, >>> >>> We achieve these performances because we've developed our own graph >>> database specifically for this task and connected it to the OpenLR >>> implementation on github. >>> We're using it with TomTom's Multinet of a large portion of the world >>> (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per >>> second (it depends on the density of the graph in the area). >>> >>> If you are interested, we could offer a simple REST APIs for decoding >>> OpenLR in batches. >>> >>> Can you share some information about your project? >>> >>> best, >>> >>> antonio >>> >>> On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 ***@***.***> >>> wrote: >>> >>> > Hi Antonio >>> > >>> > Yes I am interested, could please share more details >>> > >>> > Regards >>> > Sudhahar >>> > >>> > >>> > >>> > On Thu, Mar 25, 2021, 5:14 PM antonio di noto ***@***.***> >>> > wrote: >>> > >>> > > Hello @sudhaharv77 <https://github.com/sudhaharv77>, >>> > > we have a very fast OpenLR decoder implementation (6k-50k >>> identifiers per >>> > > second on a 4 core machine with 16GB of RAM). If you are still >>> > interested, >>> > > you can contact me. >>> > > >>> > > Best, >>> > > antonio >>> > > >>> > > Hi stephencurran, >>> > > >>> > > Thanks again, >>> > > >>> > > Sorry for the late reply, i understand that openlr do not have >>> in-memory >>> > > implementation >>> > > is it any road map to implement in-memory in future? >>> > > >>> > > Can you suggest instead Java, is there any other technologies to >>> process >>> > > the data faster with Openlr. >>> > > >>> > > Regards >>> > > Sudhahar >>> > > >>> > > — >>> > > You are receiving this because you were mentioned. >>> > > Reply to this email directly, view it on GitHub >>> > > < >>> > >>> #33 (comment) >>> > >, >>> > > or unsubscribe >>> > > < >>> > >>> https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q >>> > > >>> > > . >>> > > >>> > >>> > — >>> > You are receiving this because you commented. >>> > Reply to this email directly, view it on GitHub >>> > < >>> #33 (comment) >>> >, >>> > or unsubscribe >>> > < >>> https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q >>> > >>> > . >>> > >>> >>> >>> -- >>> Antonio Di Noto, CTO >>> Duel S.p.A. >>> via del Poggio Laurentino, 66 - 00144 Roma >>> Tel. +39 06.5195951 >>> Linkedin http://it.linkedin.com/in/antoniodinoto/ >>> http://www.dueltv.com >>> Please consider the environment before printing this e-mail >>> >>> — >>> You are receiving this because you were mentioned. >>> Reply to this email directly, view it on GitHub >>> < #33 (comment) >, >>> or unsubscribe >>> < https://github.com/notifications/unsubscribe-auth/AHZBLFGWQ6U4CZUNFTMJVBTTFMG2VANCNFSM4JVW3L7Q > >>> . >>> >> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVO23RDO5BTUXCYQH7VUJLTKVECDANCNFSM4JVW3L7Q> .

-- Antonio Di Noto, CTO Duel S.p.A. via del Poggio Laurentino, 66 - 00144 Roma Tel. +39 06.5195951 Linkedin http://it.linkedin.com/in/antoniodinoto/ http://www.dueltv.com Please consider the environment before printing this e-mail