Opened 11 years ago

Last modified 10 years ago

#5597 new defect

Pidgin calls fsync far too often.

Reported by: neuron Owned by: seanegan
Milestone: Patches welcome Component: libpurple
Version: 2.4.1 Keywords: io fsync ext3
Cc: aagaande@…, elb@…, elreydetodo

Description

Pidgin does a fsync on every purple_util_write_data_to_file_absolute, which seems to be called quite a lot. When you call fsync on an ext3 filesystem (which is used by most distributions) it not only sync's the current file, but also the whole buffer memory, which can cause serius latency problems.

There should be no reason why you need to issue an fsync before closing the file. If absolutely needed then please only fsync buddy list updates, not things like chat logs/buddy icons changing.

This issue is also posted here: https://bugs.launchpad.net/ubuntu/+source/pidgin/+bug/197254 And quite possibly here as well : http://developer.pidgin.im/ticket/37

I used latencytop to locate what was causing io problems for me (I'm on an encrypted raid, so I'm seeing this issue very easily).

Change History (11)

comment:1 Changed 11 years ago by Sim-on

  • Component changed from unclassified to pidgin (gtk)
  • Owner changed from lschiere to seanegan

comment:2 Changed 11 years ago by QuLogic

  • Component changed from pidgin (gtk) to libpurple

comment:3 Changed 11 years ago by nosnilmot

  • Cc elb@… added

This was added in 8f4d756b61b5688ce770c469c3a7095dbe6c271d in response to ticket #1102 - personally I think if people use XFS they should expect filesystem corruption on unclean shutdown (and I heard a rumor it might be fixed in Linux 2.6.22).

Maybe we can #ifdef _WIN32 this now?

comment:4 Changed 11 years ago by elb

We probably should limit this to configuration files, if it's being called for other purposes. For configuration files (which I suspect is actually the problem -- we write a lot of stuff way too often!), I don't know exactly what to do. I agree that the corruption problem is, in general, a filesystem issue, and not something we should have to work around. However, it does seem to hit a lot of people (for whatever reason).

Restricting the more draconian checks to win32 seems fine to me. The bare fsync, however, probably should stay. A better long-term solution might be to move the really chatty things (caches of buddy state information like remote aliases, window positions, etc.) to separate files which are designed to tolerate corruption more ably.

comment:5 follow-up: Changed 11 years ago by neuron

I really don't see why you'd need to fsync anything but the buddy list, if you have an unclean shutdown you can expect SOME issues, I doubt anyone would be too upset about missing 1 entry from a chatlog/missing a cached buddylist icon. And as a general rule, if your using XFS, you should have a ups, the people who use XFS should be aware of this already, xfs DOES have issues with unclean shutdowns.

It's not the job of a instant messenger to prevent filesystem corruption, especially when it comes with a big performance hit for the overall system.

comment:6 in reply to: ↑ 5 ; follow-up: Changed 11 years ago by elb

Replying to neuron:

I really don't see why you'd need to fsync anything but the buddy list, if you have an unclean shutdown you can expect SOME issues, I doubt anyone would be too upset about missing 1 entry from a chatlog/missing a cached buddylist icon. And as a general rule, if your using XFS, you should have a ups, the people who use XFS should be aware of this already, xfs DOES have issues with unclean shutdowns.

Well, not just the buddy list, but yes, that's more or less what I was suggesting. User preferences in general should be treated with some care.

It's not the job of a instant messenger to prevent filesystem corruption, especially when it comes with a big performance hit for the overall system.

Agreed. However, most people don't seem to see the performance hit that you are; keep in mind that most desktop systems are extremely lightly loaded and I/O idle. The current behavior was put in to solve a real problem (our responsibility or not). Now that we are aware that people have a problem with the current behavior, we can start looking for ways to mitigate it.

comment:7 in reply to: ↑ 6 ; follow-up: Changed 11 years ago by neuron

Replying to elb:

Replying to neuron:

I really don't see why you'd need to fsync anything but the buddy list, if you have an unclean shutdown you can expect SOME issues, I doubt anyone would be too upset about missing 1 entry from a chatlog/missing a cached buddylist icon. And as a general rule, if your using XFS, you should have a ups, the people who use XFS should be aware of this already, xfs DOES have issues with unclean shutdowns.

Well, not just the buddy list, but yes, that's more or less what I was suggesting. User preferences in general should be treated with some care.

Agreed.

It's not the job of a instant messenger to prevent filesystem corruption, especially when it comes with a big performance hit for the overall system.

Agreed. However, most people don't seem to see the performance hit that you are; keep in mind that most desktop systems are extremely lightly loaded and I/O idle. The current behavior was put in to solve a real problem (our responsibility or not). Now that we are aware that people have a problem with the current behavior, we can start looking for ways to mitigate it.

I'm not talking about idle performance, if I do a file transfer while talking to people over pidgin, I can see a noticeable performance hit on the transfer.

As you suggested, sync'ing only preferances/buddy lists, and not updating that unless it needs updates, would be a very good solution.

comment:8 in reply to: ↑ 7 ; follow-up: Changed 11 years ago by elb

Replying to neuron:

I'm not talking about idle performance, if I do a file transfer while talking to people over pidgin, I can see a noticeable performance hit on the transfer.

File transfers do not use this method, and should not be fsync'ing. You are seeing a different issue, with file transfer, I believe. We update the file transfer dialog every time a read succeeds (or did last time I was aware), which is very chatty and resource-intensive. It needs some hysteresis, is all. This is a different bug.

comment:9 in reply to: ↑ 8 Changed 11 years ago by neuron

Replying to elb:

Replying to neuron:

I'm not talking about idle performance, if I do a file transfer while talking to people over pidgin, I can see a noticeable performance hit on the transfer.

File transfers do not use this method, and should not be fsync'ing. You are seeing a different issue, with file transfer, I believe. We update the file transfer dialog every time a read succeeds (or did last time I was aware), which is very chatty and resource-intensive. It needs some hysteresis, is all. This is a different bug.

Your reading me wrong, it's when I'm doing a file transfer seperate from pidgin, if I have pidgin running at the same time and it keeps calling fsync it causes huge latency issues.

comment:10 Changed 10 years ago by bernmeister

Should this ticket be moved to Patches Welcome? Does this "issue" still exist?

comment:11 Changed 10 years ago by darkrain42

  • Milestone set to Patches welcome

I think more recent kernels have fixed some of the ext3 issues with flushing all I/O, but I'm not sure about that.

Note: See TracTickets for help on using tickets.
All information, including names and email addresses, entered onto this website or sent to mailing lists affiliated with this website will be public. Do not post confidential information, especially passwords!