On Mon, 2006-08-14 at 17:24 -0400, Adam Zey wrote:
> Robert Cummings wrote:
> > On Mon, 2006-08-14 at 13:16 -0400, Adam Zey wrote:
> >> I was writing a shell script in PHP (4.4.2) that dealt with a rather
> >> large array. To figure out what I needed the new memory limit to be, I
> >> did a memory_get_usage() at the end of my script, and came up with about
> >> 5.5MB. I then set the memory limit to 8MB.
> >>
> >> When I tried to run it, the script ran out of memory on the line:
> >>
> >> $numwords = count($words);
> >>
> >> However, when I switched to simply incrementing $numwords every time I
> >> added an element to $words, the memory limit of 8MB was fine.
> >>
> >> So my question is, if PHP does copy-on-write, why does PHP make a copy
> >> of an array when you use count() on it, which should NOT be modifying
> >> the array?
> >
> > For some reason the memory_get_usage() function wouldn't appear in my
> > PHP compilation even after using the --enable-memory-limit flag, and
> > rather than dig very deep, I whipped up the following script to test
> > your issue (under PHP 4.2.2):
> >
> > <?php
> >
> > //echo 'Mem Usage: '.memory_get_usage()."\n";
> >
> > $foo = array();
> >
> > for( $i = 0; $i < 10000000; $i++ )
> > {
> > $foo[$i] = $i;
> > }
> >
> > echo 'Created big array!'."\n";
> > sleep( 10 );
> >
> > //echo 'Mem Usage: '.memory_get_usage()."\n";
> >
> > $numEntries = count( $foo );
> >
> > echo 'Counted big array!'."\n";
> > sleep( 10 );
> >
> > //echo 'Mem Usage: '.memory_get_usage()."\n";
> > ?>
> >
> > Using the following command:
> >
> > watch -n 0 'ps awxu | grep foo.php | grep -v grep'
> >
> > I got the following snapshots during the two sleep steps:
> >
> > rob 16018 66.7 44.7 935084 928684 pts/7 S+ 17:11
> > 0:18 /usr/local/bin/php -qC ./foo.php
> >
> > rob 16018 43.9 44.7 935084 928684 pts/7 S+ 17:11
> > 0:18 /usr/local/bin/php -qC ./foo.php
> >
> > Which indicated no change from the 935 megs of memory already allocated
> > before the count().
> >
> > You've either encountered a bug in your version, or a confounding
> > variable :)
> >
> > Cheers,
> > Rob.
>
> That's the thing, count only creates a duplicate of the array (or
> consumes massive amounts of memory) *during* the call of count(). It
> frees the memory right after. The problem is that if you've got a 2MB
> array, you can't call count() on it because the temporarily increased
> memory usage will break the 4MB memory limit.
>
> Here's a better test case:
>
> 1) Ensure the memory limit is enabled and set to 4MB
> 2) Create an array that is 3MB in size
> 3) Try to call count() on that array
>
> With PHP 4.4.2, this will fail, because count will try to copy the array
> (or do something else that consumes a lot of memory). If you increase
> the memory limit to compensate, the memory usage goes back down
> immediately after the count call. For this reason, memory_get_usage()
> will never show the extra memory usage; it's allocated and freed
> entirely during the count() call.
When I ran the original test I was watching the process, it generally
takes more than a second on most system to allocate several hundred
megabytes which would have exposed your problem as a spike. At any
rate...
I figured out my problem with the recompile and then ran the script with
appropriate settings. On the first run I determined the memory required
and then for the second run I set the memory to an amount very close to
what was used. Here is second script:
#!/usr/local/bin/php -qC
<?php
ini_set( 'memory_limit', '627150412' );
echo 'Mem Usage: '.memory_get_usage()."\n";
$foo = array();
for( $i = 0; $i < 10000000; $i++ )
{
$foo[$i] = $i;
}
echo 'Created big array!'."\n";
echo 'Mem Usage: '.memory_get_usage()."\n";
$numEntries = count( $foo );
echo 'Counted big array!'."\n";
echo 'Mem Usage: '.memory_get_usage()."\n";
?>
Following I the output:
Mem Usage: 41296
Created big array!
Mem Usage: 627150256
Counted big array!
Mem Usage: 627150320
Changing the memory limit from '627150412' to '627150212' result sin the
expected memory limit exception:
Mem Usage: 41296
<br />
<b>Fatal error</b>: Allowed memory size of 627150212 bytes exhausted
(tried to allocate 12 bytes) in <b>/home/suds/foo.php</b> on line
<b>10</b><br />
So I'm not experiencing your memory issue since due to the the immense
size of the array I'm creating it would certainly show if a copy was
performed. That said (and maybe this is related to the recent memory
thread on internals that I sort of skipped over), I'm very surprised
that while I allowed '627150412' bytes for memory, that the PHP process
climbed to 900+ megs. It seems as though it doesn't account for it's own
usage of memory, which is extremely misleading. Admittedly this kind of
allocation on a production web site would normally be considered
ludicrous, it still strikes me that the memory_limit ini setting is
somewhat misleading -- in this case by about 30%.
Cheers,
Rob.
--
.------------------------------------------------------------.
| InterJinn Application Framework - http://www.interjinn.com |
:------------------------------------------------------------:
| An application and templating framework for PHP. Boasting |
| a powerful, scalable system for accessing system services |
| such as forms, properties, sessions, and caches. InterJinn |
| also provides an extremely flexible architecture for |
| creating re-usable components quickly and easily. |
`------------------------------------------------------------'
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
