FlushFileBuffers function (fileapi.h)

Flushes the buffers of a specified file and causes all buffered data to be written to a file.

Syntax

Parameters

A handle to the open file.

The file handle must have the GENERIC_WRITE access right. For more information, see File Security and Access Rights.

If hFile is a handle to a communications device, the function only flushes the transmit buffer.

If hFile is a handle to the server end of a named pipe, the function does not return until the client has read all buffered data from the pipe.

Return value

If the function succeeds, the return value is nonzero.

If the function fails, the return value is zero. To get extended error information, call GetLastError.

The function fails if hFile is a handle to the console output. That is because the console output is not buffered. The function returns FALSE, and GetLastError returns ERROR_INVALID_HANDLE.

Remarks

Typically the WriteFile and WriteFileEx functions write data to an internal buffer that the operating system writes to a disk or communication pipe on a regular basis. The FlushFileBuffers function writes all the buffered information for a specified file to the device or pipe.

Due to disk caching interactions within the system, the FlushFileBuffers function can be inefficient when used after every write to a disk drive device when many writes are being performed separately. If an application is performing multiple writes to disk and also needs to ensure critical data is written to persistent media, the application should use unbuffered I/O instead of frequently calling FlushFileBuffers. To open a file for unbuffered I/O, call the CreateFile function with the FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH flags. This prevents the file contents from being cached and flushes the metadata to disk with each write. For more information, see CreateFile.

To flush all open files on a volume, call FlushFileBuffers with a handle to the volume. The caller must have administrative privileges. For more information, see Running with Special Privileges.

When opening a volume with CreateFile, the lpFileName string should be the following form: \.\x: or \?\Volume<GUID>. Do not use a trailing backslash in the volume name, because that indicates the root directory of a drive.

In WindowsВ 8 and Windows ServerВ 2012, this function is supported by the following technologies.

What does it mean to ‘flush to disk’?

Could someone explain what is meant by flushing to disk in the following context? If I am writing data to a log on a filesystem, doesn’t this mean I am putting it on disk? At what point would/should you flush a file to disk?

This suggests a design which is very simple: rather than maintain as much as possible in-memory and flush it all out to the filesystem in a panic when we run out of space, we invert that. All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means that it is transferred into the kernel’s pagecache.

2 Answers 2

All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means that it is transferred into the kernel’s pagecache.

What this means is that Kafka hands data off to the kernel with write() syscalls — at which point in time it’s visible to other processes but may or may not actually be reflected on disk and survive a reboot — but doesn’t force the kernel to rush it to disk with fsync() calls or similar (as appropriate for the OS at hand). If optimizing for throughput and not needing to guarantee that content is retrievable, this can be an appropriate decision: fsync() and its kin can be expensive calls (though by doing long contiguous writes that don’t require seeking, kafka minimizes the expense of its disk IO).

How can I reclaim “modified” memory for my system? Any way to flush it to disk?

My hard drive has been aging. Windows hasn’t been treating it well. My system has a «measly» 4 GB of RAM and Windows is all too happy to thrash all of the memory to swap and back. This means that before I disabled swap entirely on my system, my disk would have average response times of

10 seconds all the damn time. This ended when I disabled swap; the system is now infinitely more pleasant to use.

That is, until things predictably start vanishing under your feet all day long. Six tabs and one whole copy of Chrome crashed in the making of this question, and this is because of Windows 8’s seemingly trigger happy out of memory killer.

Of course the devil is in the details. While I only have «1.9 GB» in use, I only have «1.5 GB» available; neither of those figures matter anyway, because «2.9/4.0 GB» are committed, so the actual memory left on my system for the purposes of the OOM killer is 1.1 GB.

RamMap has this to say about the situation:

As I read it, 700 MB of memory seemingly could stand to be flushed to disk, and is being flushed (the number seems to diminish over time), but very slowly. Is that what’s going on? How can I accelerate this process, then?

2 Answers 2

to empty the modified list. High values can be caused by older Brodcom drivers. Also don’t disable the pagefile!! Never do this.

For the most part, Windows NEEDS a swapfile. Without it, weird things will happen. Even if you have a system with 64GB of memory, most would think a swap file is not needed. However Windows itself is coded and expected to have a swap file.

You can prevent those weird problems from popping up and have no thrashing with a tiny swap file of 256MB or greater on ANY drive. It prefers a min 400MB on C: (or boot drive) for crash dumps, but that’s not needed and wont affect stability.

So instead of disabling it, just put a 256-512MB swap to ANY drive. Windows won’t thrash that. It will just put a bit of idle code here and there. If you have 4GB or less (even on a 4GB system), I’d recommend a 1GB swap to prevent out of memory errors and major instability.

Normally you can’t put a swap on a thumb drive, but some USB 3.0 thumb drives appear as a HDD vs a «removable» drive. Better yet, grab an eSATA thumb drive thats 16-32GB. then you can shove a 32GB swap on it if you like LOL! It won’t need it, but it will certainly use the swap a LOT faster VS an HDD (unless you don’t have HDDs and have SDDs)

I know I’ve seen TONS of people disable paging in Windows, and you should be able to do that with no repercussions. However, no matter how much ram you have, Windows wants SOMETHING to page. even if only 256MB

It’s not ios, linux, or OSX where swapping is 100% optional. Windows is «sort of optional» You can 100% disable paging, but then some people are plagued with issues and others have minor ones

My stance: Why take the chance. A 256MB swap isn’t going to thrash your system to death.

Better yet.. get a USB 3.0 32GB thumb drive (a FAST one.. and they are CHEAP). format it NTFS.. and DEDICATE it to Readyboost (if you use anything other than NTFS , you are limited to 4GB).

Unless you are PURE SDD or PURE SATA 3 (or SATAe — not eSATA!!) with all Raid 0s, that usb 3.0 RB will make your system FLY. It will cache 32GB of cache data that can be read FAR faster than any SATA 3 drives. NO Sata 3 drive can even come close to the 6Gps cap.. not even the SDDs.. and USB is 5Gps/sec.. I’ve yet to see any Sata3 HDD outperform a fast USB 3.0 thumb drive

Windows 8 is best for this because of the Fast Start up (Fast Start up just logs you out then hibernates the system). why is this good? a True shut down or restart erases the RB cache.. and Caching 32GB of data takes TIME, and HDD time. but once it’s done.. you’re GOLDEN. STandby, Hybernate, or Fast Start up KEEPS the RB cache 🙂 So you system will just fly (until you do a full restart or true shutdown)

RB isn’t RAM, and it’s not a swap file, but it is cache.. and 32GB of cached data will also keep windows from gobbling up RAM for cache and having to clear it when needed. And like I said, some thumb drives, Win 8 treats as an HDD and you can put a swap there. then you get a swap with ZERO hard drive thrash 🙂 Great set up: 4 GB usb 3.0 drive as a 4GB swap, then a 32GB usb thumb for 32GB of HDD cache. Again, RB is useless if your system is ALL SDDs or ALL drives are raided to the point they are faster than RB. And windows isn’t stupid. if all your drives are faster than the RB device, it will NOT let you enable RB 🙂

How to flush the disk read cache under Windows? [duplicate]

In order to be able to so some benchmarks I need to cleanup Windows disk read cache. How can I do this?

In fact I want to compare if loading a big Unicode file (UTF-8 or UTF-16) from disk is faster or not, considering that in memory I do keep UTF-16.

I know that it should be no significant difference but in order to benchmark it I need to be sure that that file is not cached — I need to see if size on disk has more or less impact than decoding the file.

3 Answers 3

Someone from Microsoft helped me out with this a few years ago (back in the days of Windows XP) and gave me a solution that met my needs at the time. I already had two drives in my machine, and what I needed to test was already on the D drive, and fortunately I didn’t need to keep any files open on my D drive when I wanted to flush the cache. From the Windows Disk Management interface, I could change my D: drive letter to another letter, and then back again, and the D drive would perform as if had a cold cache in the OS.

AFAIK, it’s unfortunately not possible to discard the read cache under Windows. I spent some time looking into this some years ago, and only found out how to flush the write cache.

As I see it, you have three options, unless somebody else has found some magic:

1. If possible, do your read file I/O in unbuffered mode.
2. Each time you want to benchmark, create a new copy of the test data specifying unbuffered mode when creating the new copy (this should keep the copy out of read cache, but I haven’t tested).
3. Allocate enough memory that windows has to discard the disk cache (ugh!).

EDIT: it is indeed possible to flush the read cache, at least on Vista and later: Disable or flush page cache on Windows. It requires a call to the undocumented NtSetSystemInformation NT API, though. Also, for a single file, read cache can be flushed simply by opening the file with FILE_FLAG_NO_BUFFERING specified and closing the handle again.

Flush-to-disk option #18

Comments

temp-impl commented Sep 15, 2016

In Windows 10 and Windows Server 2012 R2, StreamWriter.Flush won’t write the data to disk immediately.

You can know this by running under code and watching the file size through «dir» command.

We can write to disk immediately by touching the file like this.

Could you please improve «buffered:false» or add new option to do this?

P.S.
I’m using Serilog.Sinks.RollingFile for windows service logging, and collecting the logs by Microsoft Operations Management Suite (OMS). I found OMS log collection is not working good, because above.

The text was updated successfully, but these errors were encountered:

nblumhardt commented Sep 18, 2016

Thank for the note. Full sync-to-disk is very, very slow (order of milliseconds). I’ve been hesitant to add the option because in most apps it’ll be unacceptable. The current behavior is to flush through to the OS page cache, which, while it won’t protect from data loss in a power outage (very little can), deals with process crashes and the like.

In your scenario, would the perf hit be acceptable? Is the concern data loss, or is OMS the primary driver for needing this?

temp-impl commented Sep 20, 2016 •

This is my problem detail.

• OMS doesn’t collect logs that file size is not increased.
• Serilog.Sinks.RollingFile doesn’t write log to disk until it’s rolling, because OS page cache is large.
  The log file size of a day is about 300KB from 600KB.
• I want to collect logs for monitoring the windows service. OMS can make alert if there are no logs during some time.

I found another way to write to disk almost immediately.
In this way, the buffer size is reduced to 1KB. Tested in Windows 10 and Windows Server 2012 R2.

It’s very helpful if my problem will be solved by the library, but I know it is difficult problem for the library.

pakrym commented Oct 3, 2016

@nblumhardt we have the same issue while using Serilog for Azure AppServices logging, Log Stream feature picks up changes only when logs file size is increased.

Azure AppServices file logging is intended as temporary thing that you turn on to debug you application live, so we are somewhat fine with taking a performance hit although it would be better to flush events in batches with a short timeout rather then each time.

Other solution is to have timer and open/close file for reading in parallel which will also force OS to flush the data/update attributes.

nblumhardt commented Oct 3, 2016

@pakrym thanks for the info, sounds like this is something we’ll need to accommodate.

A simple wrapper that closed/reopened the file by disposing/re-creating the sink on a timer could be implemented atop what’s there now, but doesn’t seem like a long-term option.