Xalan 2.7.1 - File Handle Limitation?

[basZero]

Hi,

I'm observing the following:
my Java 1.8 application (web app on Tomcat 8) is running on Ubuntu 16 under
a user that has a ulimit of 65000.

A certain application logic within my Java app is looping through 80000
different files. When this loop is performed in Java code, there is no
issue, all data can be processed as expected.

When I run the loop via XSLT by using the document() function (Xalan 2.7)
to open the files, it stops opening files after approx. 32000, depending on
the current load on our system. By stopping I mean that XSL can not access
the content of the requested file anymore.

I'm wondering if there is a limitation besides the ulimit of Ubuntu within
my technology stack here?

Any help regarding Xalan would be highly appreciated.
Thanks
Cheers, Bas

[Gary Gregory]

Post by basZero
Hi,
my Java 1.8 application (web app on Tomcat 8) is running on Ubuntu 16
under a user that has a ulimit of 65000.
A certain application logic within my Java app is looping through 80000
different files. When this loop is performed in Java code, there is no
issue, all data can be processed as expected.
When I run the loop via XSLT by using the document() function (Xalan 2.7)
to open the files, it stops opening files after approx. 32000, depending on
the current load on our system. By stopping I mean that XSL can not access
the content of the requested file anymore.

Do you get an exception? Does your app hang?

Gary

Post by basZero
I'm wondering if there is a limitation besides the ulimit of Ubuntu within
my technology stack here?
Any help regarding Xalan would be highly appreciated.
Thanks
Cheers, Bas

--
E-Mail: ***@gmail.com | ***@apache.org
Java Persistence with Hibernate, Second Edition
<https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygregory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>

<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1617290459>
JUnit in Action, Second Edition
<https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygregory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22>

<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182021>
Spring Batch in Action
<https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+in+Action>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182951>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

[Balz Schreier]

No it doesnt hang, no exception

Post by Gary Gregory

Hi,
my Java 1.8 application (web app on Tomcat 8) is running on Ubuntu 16 under a user that has a ulimit of 65000.
A certain application logic within my Java app is looping through 80000 different files. When this loop is performed in Java code, there is no issue, all data can be processed as expected.
When I run the loop via XSLT by using the document() function (Xalan 2.7) to open the files, it stops opening files after approx. 32000, depending on the current load on our system. By stopping I mean that XSL can not access the content of the requested file anymore.

Do you get an exception? Does your app hang?
Gary

I'm wondering if there is a limitation besides the ulimit of Ubuntu within my technology stack here?
Any help regarding Xalan would be highly appreciated.
Thanks
Cheers, Bas

--
Java Persistence with Hibernate, Second Edition
JUnit in Action, Second Edition
Spring Batch in Action
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

[Scott Boag]

From my deep memory, I think there is indeed a limitation imposed by a 16-bit space embedded doc id in the node id, or something like that. Joe may remember more details.

Sent from Lotus Traveler


basZero --- Xalan 2.7.1 - File Handle Limitation? ---
From:"basZero" <***@gmail.com>To:***@xalan.apache.orgDate:Tue, Nov 1, 2016 12:54Subject:Xalan 2.7.1 - File Handle Limitation?

Hi,

I'm observing the following:
my Java 1.8 application (web app on Tomcat 8) is running on Ubuntu 16 under a user that has a ulimit of 65000.


A certain application logic within my Java app is looping through 80000 different files. When this loop is performed in Java code, there is no issue, all data can be processed as expected.

When I run the loop via XSLT by using the document() function (Xalan 2.7) to open the files, it stops opening files after approx. 32000, depending on the current load on our system. By stopping I mean that XSL can not access the content of the requested file anymore.

I'm wondering if there is a limitation besides the ulimit of Ubuntu within my technology stack here?

Any help regarding Xalan would be highly appreciated.
Thanks
Cheers, Bas

[Joseph Kesselman]

<font face="Verdana,Arial,Helvetica,sans-serif" size="2"> <span>As I remember it -- I'd have to re-read the code -- the Document Table Model used a 32-bit node id, split into two bitfields identifying documents and node within the document. This was parameterized, so in theory it would be possible to support more documents at the cost of reducing the maximum size of each -- but I don't remember whether that was a compile-time decision or if it could be adjusted when Xalan was started.<br><br>If you only need to scan those documents sequentially rather than having them all in memory at once, Xalan has (had?) an extension that discarded each one when the next was retrieved from the document() call. I'm far from certain, but that *might* also reuse document IDs, and thus solve your problem.<br><br>IBM's XTQHP optimizing compiler for XSLT used a different internal API to access documents, and hence avoided this field-width limitation -- but as far as I know that processor is still available only as the XSLT Feature for WebSphere Traditional and hasn't been made available any other way. Last I checked, IBM Java still shipped with a version of Xalan (though at the time, it was a significantly more recent version of Xalan than the one Sun was packaging with their JRE).<br><br>Hope that was at least somewhat useful...<br><br>______________________________________<br>"Your data is important to us. Please stay on the port, and your connection will be accepted by the next available thread."</span><br><br><font color="#000000" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><font color="#990099">-----Scott Boag/Cambridge/IBM wrote: -----</font></font><div class="iNotesHistory" style="padding-left:5px;"><font color="#000000" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><div style="padding-right:0px;padding-left:5px;border-left:solid black 2px;">To: <a target="_blank" href="mailto:***@xalan.apache.org">***@xalan.apache.org</a><br>From: Scott Boag/Cambridge/IBM<br>Date: 11/01/2016 04:48PM<br>Cc: <a target="_blank" href="mailto:***@xalan.apache.org">***@xalan.apache.org</a>, Joseph Kesselman/Watson/***@lotus<br>Subject: Re: Xalan 2.7.1 - File Handle Limitation?<br><br><font style="white-space:pre-wrap;font-family: Helvetica Neue, Helvetica, Arial, sans-serif;margin: 1em 0;">From my deep memory, I think there is indeed a limitation imposed by a 16-bit space embedded doc id in the node id, or something like that.&nbsp; Joe may remember more details. <br><br>Sent from Lotus Traveler</font><br><br><div class="domino-section" dir="ltr"><div class="domino-section-head"><span class="domino-section-title"><font color="#424282">basZero --- Xalan 2.7.1 - File Handle Limitation? --- </font></span></div><div class="domino-section-body"><br><table cellpadding="0" cellspacing="0" border="0" width="100%"><tbody><tr valign="top"><td style="width: 96px;" width="1%"><font color="#5F5F5F" size="2">From:</font></td><td style="width: auto;" width="100%"><font size="2">"basZero" &lt;<a target="_blank" href="mailto:***@gmail.com">***@gmail.com</a>&gt;</font></td></tr><tr valign="top"><td style="width: 96px;" width="1%"><font color="#5F5F5F" size="2">To:</font></td><td style="width: auto;" width="100%"><font size="2"><a target="_blank" href="mailto:***@xalan.apache.org">***@xalan.apache.org</a></font></td></tr><tr valign="top"><td style="width: 96px;" width="1%"><font color="#5F5F5F" size="2">Date:</font></td><td style="width: auto;" width="100%"><font size="2">Tue, Nov 1, 2016 12:54</font></td></tr><tr valign="top"><td style="width: 96px;" width="1%"><font color="#5F5F5F" size="2">Subject:</font></td><td style="width: auto;" width="100%"><font size="2">Xalan 2.7.1 - File Handle Limitation?</font></td></tr></tbody></table><hr style="" align="left" size="2" width="100%"><br></div><div dir="ltr">Hi,<div><br></div><div><div>I'm observing the following:&nbsp;</div><div>my Java 1.8 application (web app on Tomcat 8) is running on Ubuntu 16 under a user that has a ulimit of 65000.<br></div><div><br></div><div>A certain application logic within my Java app is looping through 80000 different files. When this loop is performed in Java code, there is no issue, all data can be processed as expected.&nbsp;</div><div><br></div><div>When I run the loop via XSLT by using the document() function (Xalan 2.7) to open the files, it stops opening files after approx. 32000, depending on the current load on our system. By stopping I mean that XSL can not access the content of the requested file anymore.</div><div><br></div><div>I'm wondering if there is a limitation besides the ulimit of Ubuntu within my technology stack here?<br><br>Any help regarding Xalan would be highly appreciated.</div><div>Thanks</div><div>Cheers, Bas</div><div><br></div></div></div></div></div></font></div></font><BR>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-***@xalan.apache.org
For additional commands, e-mail: dev-***@xalan.apache.org

[basZero]

Thanks Scott and Joseph,
your comments support my impression which I had when analyzing my issue.
Is there any suggestion from your side what I should do? It's a productive
code for an e-Commerce site so it would be great to have this fully fixed.

Thanks again.

Post by Joseph Kesselman
As I remember it -- I'd have to re-read the code -- the Document Table
Model used a 32-bit node id, split into two bitfields identifying documents
and node within the document. This was parameterized, so in theory it would
be possible to support more documents at the cost of reducing the maximum
size of each -- but I don't remember whether that was a compile-time
decision or if it could be adjusted when Xalan was started.
If you only need to scan those documents sequentially rather than having
them all in memory at once, Xalan has (had?) an extension that discarded
each one when the next was retrieved from the document() call. I'm far from
certain, but that *might* also reuse document IDs, and thus solve your
problem.
IBM's XTQHP optimizing compiler for XSLT used a different internal API to
access documents, and hence avoided this field-width limitation -- but as
far as I know that processor is still available only as the XSLT Feature
for WebSphere Traditional and hasn't been made available any other way.
Last I checked, IBM Java still shipped with a version of Xalan (though at
the time, it was a significantly more recent version of Xalan than the one
Sun was packaging with their JRE).
Hope that was at least somewhat useful...
______________________________________
"Your data is important to us. Please stay on the port, and your
connection will be accepted by the next available thread."
-----Scott Boag/Cambridge/IBM wrote: -----
From: Scott Boag/Cambridge/IBM
Date: 11/01/2016 04:48PM
Subject: Re: Xalan 2.7.1 - File Handle Limitation?
From my deep memory, I think there is indeed a limitation imposed by a
16-bit space embedded doc id in the node id, or something like that. Joe
may remember more details.
Sent from Lotus Traveler
basZero --- Xalan 2.7.1 - File Handle Limitation? ---
Date: Tue, Nov 1, 2016 12:54
Subject: Xalan 2.7.1 - File Handle Limitation?
------------------------------
Hi,
my Java 1.8 application (web app on Tomcat 8) is running on Ubuntu 16
under a user that has a ulimit of 65000.
A certain application logic within my Java app is looping through 80000
different files. When this loop is performed in Java code, there is no
issue, all data can be processed as expected.
When I run the loop via XSLT by using the document() function (Xalan 2.7)
to open the files, it stops opening files after approx. 32000, depending on
the current load on our system. By stopping I mean that XSL can not access
the content of the requested file anymore.
I'm wondering if there is a limitation besides the ulimit of Ubuntu within
my technology stack here?
Any help regarding Xalan would be highly appreciated.
Thanks
Cheers, Bas
--------------------------------------------------------------------- To