How to Streaming Buffered PHP Output in Zend Framework

How to Streaming Buffered PHP Output in Zend Framework

Streaming Buffered PHP Output in Zend Framework, step by step.

I was recently working on a feature that required me to retrieve large amounts of data (7 figure document counts) from Elasticsearch, modify the data and then output a CSV file. This feature needed to be performant and not have an adverse effect on the memory usage of the application's EC2 instance.

An original implementation of the feature retrieved all documents, stored them in memory and then created a file on disk that would be served to the user. This resulted in memory exhaustion, as well as taking up large amounts of disk space on production instances.

This pseudocode represents the initial implementation we had:

class ExportController() {
    public function exportAction(): Stream {
        $exportQuery = $this->params()->fromPost('query');
        $this->dataExporter->export($exportQuery);

    $response        = $this->getResponse();
    $responseHeaders = $response
        ->getHeaders()
        ->addHeaders([
            'Content-Description'       => 'File Transfer',
            'Content-Type'              => 'text/csv',
            'Content-Transfer-Encoding' => 'binary',
            'Expires'                   => 0,
            'Cache-Control'             => 'must-revalidate',
            'Pragma'                    => 'public',
        ]);
    $response->setHeaders($responseHeaders);
    $response->setContent($fileContents);
    return $response;
}

}

class DataExporter { public function export($exportQuery) { // Get all of the results $results = $this->elasticsearchService->getData($exportQuery);

    // Iterate through data and output to CSV file
    $filePointer = fopen('/tmp/data.csv', 'wb');
    foreach ($results as $result) {
        fputcsv($filePointer, $result, ',');
    }
}

}

fputcsv takes a resource as its first parameter, including a resource that points to a PHP I/O stream such as php://output. By replacing /tmp/data.csv with php://output we removed our need to create CSV files on the server.

Now that we're writing the data straight to the output stream, we need to send the correct headers before we flush the buffer:

class ExportController() {
    public function exportAction(): Stream {
        $exportQuery = $this->params()->fromPost('query');
        $this->dataExporter->export($exportQuery);
    }
}

class DataExporter { public function export($exportQuery): void { // Get all of the results $results = $this->elasticsearchService->getData($exportQuery);

    // Iterate through data and output to CSV file
    ob_start();
    $outputPointer = fopen('php://output', 'wb');
    foreach ($results as $result) {
        fputcsv($outputPointer, $result, ',');
    }

    // Close our file pointer, send the file headers and flush our output buffer
    fclose($filePointer);
    header('Content-Description: File Transfer');
    header('Content-Type: text/csv');
    header('Content-Transfer-Encoding: binary');
    header('Expires: 0');
    header('Cache-Control: must-revalidate');
    header('Pragma: public');
    ob_end_flush();
    exit();
}

}

We use exit() to avoid a headers already sent error as Zend Framework will try to send a response, and more headers, to the user.

If we were to write a test for the export method, we'd swiftly run into the issue where our usage of exit() ends our PHPUnit process. Short of monkey patching the exit() function, there's not much we can do about this. Other than removing exit() that is. Instead of flushing the buffer in our export method, we can instead create a Stream response and return that to our controller. The controller will then return the Stream to Zend Framework which will send it to our user.

class ExportController() {
    public function exportAction(): Stream {
        $exportQuery = $this->params()->fromPost('query');
        return $this->dataExporter->export($exportQuery);
    }
}

class DataExporter { public function export($exportQuery): Stream { // Get all of the results $results = $this->elasticsearchService->getData($exportQuery);

    // Create our Stream
    $outputPointer = fopen('php://output', 'wb');
    $stream = new Stream();
    $stream->setStream($outputPointer);
    $stream->setHeaders((new Headers())->addHeaders([
        'Content-Description'       => 'File Transfer',
        'Content-Type'              => 'text/csv',
        'Content-Transfer-Encoding' => 'binary',
        'Expires'                   => '0',
        'Cache-Control'             => 'must-revalidate',
        'Pragma'                    => 'public',
    ]);

    // Iterate through data and output to CSV file
    foreach ($results as $result) {
        fputcsv($outputPointer, $result, ',');
    }

    return $stream;
}

}

When a response is returned by a controller, Zend MVC emits a SendResponseEvent event that is listened to by several Zend Framework listeners. A listener is available for each type of response we may return. In our case, the StreamResponseSender will be responsible for sending our response onwards.

  • Zend\Mvc\SendResponseListener\PhpEnvironmentResponseSender with a priority of -1000
  • Zend\Mvc\SendResponseListener\ConsoleResponseSender with a priority of -2000
  • Zend\Mvc\SendResponseListener\SimpleStreamResponseSender with a priority of -3000
class SimpleStreamResponseSender extends AbstractResponseSender
{
    public function sendStream(SendResponseEvent $event)
    {
        if ($event->contentSent()) {
            return $this;
        }
        $response = $event->getResponse();
        $stream   = $response->getStream();
        fpassthru($stream);
        $event->setContentSent();
    }

public function __invoke(SendResponseEvent $event)
{
    $response = $event->getResponse();
    if (! $response instanceof Stream) {
        return $this;
    }
    $this->sendHeaders($event);
    $this->sendStream($event);
    $event->stopPropagation(true);
    return $this;
}

}

Now unfortunately, fpassthru does not support resources using php://output. Attempting to use the StreamResponseSender will result in an error about an invalid resource. Replacing fpassthru with ob_end_flush is all that is required to make the SimpleStreamResponseSender work.

Now we'll need to create our own response sender to handle these Streamresponses using php://output. We also need to differentiate our responses using php://output from a standard Stream response.

class BufferedStream extends Stream {
    public function __construct($filePointer)
    {
        $this->setHeaders((new Headers())->addHeaders([
            'Content-Description'       => 'File Transfer',
            'Content-Type'              => 'text/csv',
            'Content-Transfer-Encoding' => 'binary',
            'Expires'                   => '0',
            'Cache-Control'             => 'must-revalidate',
            'Pragma'                    => 'public',
        ]));
    }
}

class BufferedStreamResponseSender extends AbstractResponseSender { public function sendStream(SendResponseEvent $event): self { if ($event->contentSent()) { return $this; }

    // fpassthru does not work with php://output so we have to flush the buffer instead
    ob_end_flush();
    $event->setContentSent();
    return $this;
}

public function __invoke(SendResponseEvent $event)
{
    $response = $event->getResponse();
    if (! $response instanceof BufferedStream) {
        return $this;
    }

    $this->sendHeaders($event);
    $this->sendStream($event);
    $event->stopPropagation(true);
    return $this;
}

}

Now we have our listener and our new BufferedStream, we simply need to update our DataExporter to use the correct response and attach our listener to the SendResponseEvent.

class ExportController() {
    public function exportAction(): BufferedStream {
        $exportQuery = $this->params()->fromPost('query');
        return $this->dataExporter->export($exportQuery);
    }
}

class DataExporter { public function export($exportQuery): BufferedStream { // Get all of the results $results = $this->elasticsearchService->getData($exportQuery);

    // Create our Stream
    $outputPointer = fopen('php://output', 'wb');
    $stream = new BufferedStream();
    $stream->setStream($outputPointer);

    // Iterate through data and output to CSV file
    foreach ($results as $result) {
        fputcsv($outputPointer, $result, ',');
    }

    return $stream;
}

}

class Module implements ConfigProviderInterface { /** * @inheritdoc */ public function getConfig() { return include DIR . '/../config/module.config.php'; }

/**
 * @inheritdoc
 */
public function onBootstrap(EventInterface $e)
{
    /** @var EventManager $eventManager */
    $eventManager  = $e->getApplication()->getEventManager();
    $eventManager->attach(SendResponseEvent::EVENT_SEND_RESPONSE, new BufferedStreamResponseSender(), 1);
}

}

We've now removed any need for processing data in to an intermediary

file. Data is processed and then streamed straight to PHP's buffered

output.

Elasticsearch offers the Scroll API for processing large amounts of data. It operates in a similar way to a cursor on a traditional database. By replacing our single getData call with a loop utilising the Scroll API, we can minimise the amount of data we have stored in memory at one time.

class DataExporter {
    public function export($exportQuery): BufferedStream
    {
        // Create our Stream
        $outputPointer = fopen('php://output', 'wb');
        $stream = new BufferedStream();
        $stream->setStream($outputPointer);

    // Start our scroll
    $resultSet = $this->elasticsearchService->startScroll($exportQuery);
    $this->writeDataToResource($outputPointer, $resultSet);

    // Continue the scroll till we have no further results
    while (count($resultSet->getResults()) !== 0) {
        $resultSet = $this->elasticsearchService->continueScroll($resultSet);
        $this->writeDataToResource($outputPointer, $resultSet);
    }

    return $stream;
}

private function writeDataToResource($outputPointer, ResultSet $resultSet): void
{
    $results = $resultSet->getResults();
    foreach ($results as $result) {
        fputcsv($outputPointer, $result, ',');
    }
}

}

By utilising streams and the Scroll API, we reduced our memory usage and removed our file usage completely.

Thanks for reading. If you liked this post, share it with all of your programming buddies!

Further reading

☞ PHP with Laravel for beginners - Become a Master in Laravel

☞ Laravel PHP Framework Tutorial - Full Course for Beginners (2019)

php web-development

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

PHP Web Application Development Company

As a top **PHP Web Application Development Company in USA**[https://www.dataeximit.com/php-development-services/](https://www.dataeximit.com/php-development-services/ "https://www.dataeximit.com/php-development-services/"), we at Data EximIT have...

PHP Website Development

Skenix Infotech is a leading PHP Web Development Company that helps companies get results with industry best practices. Get affordable PHP Development Services.

Hire Dedicated PHP Developer

Looking to hire affordable yet experienced PHP developers? **[Hire Dedicated PHP Developer](https://hourlydeveloper.io/hire-dedicated-php-developer/ "Hire Dedicated PHP Developer")**, who can convert your idea to reality, within the stipulated...

Advantages of Hiring PHP Developer for your Website Project

PHP - Hypertext pre-processor, a scripting language used by many people in developing web pages, but most of us are unaware even of the full form. To train someone and make them learn this whole language is as difficult and time-consuming as it is...

Hire PHP Developer and Web Developer for your Online Business

PHP is widely used open-source scripting language it helps in making dynamically easy your websites and web application. Mobiweb Technology is your best technical partner and offering you solution for any kind of website and application...