Patch Name: PHKL_28105 Patch Description: s700_800 11.00 VxFS cumulative,I/O Throttling,dirty inval Creation Date: 02/10/31 Post Date: 02/11/12 Repost: 04/05/28 The Special Installation Instructions section of the patch documentation was modified to more accurately describe the VX_NOTHROTTLE tunable. Hardware Platforms - OS Releases: s700: 11.00 s800: 11.00 Products: N/A Filesets: JournalFS.VXFS-BASE-KRN,fr=B.11.00,fa=HP-UX_B.11.00_32,v=HP JournalFS.VXFS-BASE-KRN,fr=B.11.00,fa=HP-UX_B.11.00_64,v=HP Automatic Reboot?: Yes Status: General Release Critical: Yes PHKL_28105: PANIC PHKL_27648: HANG PHKL_24905: PANIC Category Tags: defect_repair enhancement general_release critical panic halts_system Path Name: /hp-ux_patches/s700_800/11.X/PHKL_28105 Symptoms: PHKL_28105: ( SR:8606274037 CR:JAGae38116 ) A panic occurs when accessing an invalid dirty buffer in the buffer cache. This can result in a couple of different panics. Two types of panics seen so far are "panic: An invalid dirty buffer!" and a Data Page Fault in bwrite(). A sample stack trace is given below. panic: An invalid dirty buffer! panic+0x14 getnewbuf+0x6b4 allocbuf1+0x178 brealloc1+0x64 ogetblk+0x254 getblk1+0x260 vx_getblk+0x50 vx_do_read_ahead+0xa4 vx_read_ahead+0x148 vx_read1+0x38c vx_rdwr+0x98 vno_rw+0x80 read+0xfc syscall+0x6f8 $syscallrtn+0x0 PHKL_27648: ( SR:8606270211 CR:JAGae34449 ) Possible deadlock or hang during bootup on J-class machines. PHKL_27070: ( SR:8606197982 CR:JAGad67173 ) With PHKL_22312 installed, under heavy I/O, sync(2) and processes that use sync(2) (such as sync(2), bdf(1M), and login(1)) could experience long delays while dirty buffers are flushed to disk. PHKL_22312: (SR: 8606157898 CR: JAGad27228) Systems with heavy I/O to slow devices may sometimes experience slow response time for processes trying to access the respective device. PHKL_24905: ( SR: 8606204945 CR: JAGad74121 ) Mounting a VxFS filesystem may panic the system. Stack trace of the panic thread would look like: panic+0x14 report_trap_or_int_and_panic+0x84 interrupt+0x1d4 ihandler+0x928 vx_spin_lock+0x30 vx_mapiodone+0x68 vx_mapiodone_bp+0x14 biodone+0x1f0 lv_complete+0xc4 lv_terminate+0xd4 lv_parwrite_done+0x470 lv_end+0x174 biodone+0x1f0 scsi_iodone+0x44 scsi_cbfn+0x4b0 fcpdev_scsi_comp+0x94 td_io_done+0x40 td_cdb_cbfn_start+0xb0 td_isr+0x1b4 epic_isr+0x58 mp_ext_interrupt+0x318 ihandler+0x904 PHKL_19481: (SR: 5003436360 CR: JAGaa40337) If the JFS is mounted with nolog option, it can not be mounted again after unmounting it. The problem is easily reproduced. PHKL_26794: ( SR:8606233621 CR:JAGae02845 ) The 'st_nlink' field, as returned by stat(2), can be inaccurate. This only occurs on VxFS filesystems with PHKL_21210 (or any superseding patch) installed, the vx_maxlink tunable is set to a value greater than 64k-1 and the number of directories created exceeds 64k-1. PHKL_24201: (SR: 8606184447 CR: JAGad53655) VxFS directory sticky-bit not functioning according to specification and man pages. PHKL_21210: (SR: 5003451625 CR: JAGaa93198) Currently the number of subdirectories that can be created is limitted to LINK_MAX. A new tunable has now been added for VxFS (vx_maxlink) which will allow this number to be changed to any value between LINK_MAX and INT_MAX (See limits(5)). If the tunable is not set the value of the limit remains at LINK_MAX. PHKL_18796: SR: 1653264663 CR: JAGaa11143 If a VxFS file system has quotas turned on, but no quota file is present, users can get open() and creat() failures with error number '61449'. Defect Description: PHKL_28105: ( SR:8606274037 CR:JAGae38116 ) Some dirty buffers in the buffer cache get invalidated when a race occurs between VxFS mount and a block device writer (eg. fsck) operating on the device VxFS is mounting. VxFS does not prevent block device writes from being cached in the buffer cache when it flushes and then invalidates existing device buffers. This results in an open window between the flushing and invalidation of device buffers. If a dirty device buffer enters the buffer cache in this window, it can get invalidated causing a panic at a later time. Resolution: The resolution is to prevent the block device writes from adding buffers into buffer cache before VxFS mount code starts flushing/invalidation of device buffers. PHKL_27648: ( SR:8606270211 CR:JAGae34449 ) A deadlock or hang can occur on bootup with some J-class machines because of a lock synchronization issue in VxFS code. Resolution: Modified code added in PHKL_27070 that introduced the lock synchronization problem. This resolves the possible hang problem. The tunable introduced in PHKL_27070 remains the same. PHKL_27070: ( SR:8606197982 CR:JAGad67173 ) PHKL_22312 introduced a "throttling mechanism" to prevent flushes of a large file from flooding a device if the disk device could not keep up with the amount of data being flushed. The large disk I/O queue would cause slow response times for other processes trying to access the respective device. However, the throttling mechanism would cause processes flushing buffers to wait for the disk to process I/O requests before flushing more buffers, resulting in extended delays when flushing dirty buffers. Resolution: When a sync(2) is performed, a new vxfs daemon thread is created to handle the flushing of buffers on behalf of sync(2). Also, a new tunable VX_NOTHROTTLE has been created to enable or disable I/O throttling within VxFS. PHKL_22312: (SR: 8606157898 CR: JAGad27228) The method used by VxFS filesystems for writing sequential data is to asynchronously write data up to a limit and when that limit is met to delay-write remaining data. This data is flushed periodically. The adverse effect of having a large amount of delayed-write data in the buffer cache is that it can potentially flood the device with requests. Resolution: In order to slow the amount of I/O requests made to devices that cannot keep up with demand a throttling mechanism will be put in place for flushes from the buffer cache to VxFS files that will prevent a device from being overloaded. PHKL_24905: (SR: 8606204945 CR: JAGad74121 ) After getting an error on a VxFS mount, a lock could be prematurely released while an I/O to the device is still in progress. Later, when the I/O is finished, de-referencing the lock may panic the system. Resolution: Make sure that all the activities on the device are finished before deallocating the lock. PHKL_19481: (SR: 5003436360 DTS: JAGaa40337) A bug existed which, at unmount time, did not reset the nolog flag if the filesystem had been mounted with the nolog option. Any subsequent mount of this filesystem failed a sanity check and the mount. Resolution: Unset the VX_NOLOG flag from the superblock when the file system is unmounted. PHKL_26794: ( SR:8606233621 CR:JAGae02845 ) vx_maxlink was first introduced with PHKL_21210. It incorrectly allows the number of sub-directories in a directory to exceed 64k-1, the maximum value that can be displayed in the 'st_nlink' field, output by stat(2). Hence, for vx_maxlink greater than 64k-1, st_nlink overflows. Resolution: The vx_maxlink tunable is now limited to 64k-1. When tuned to values greater than this, the operating system will reset it to 64k-1. PHKL_24201: (SR: 8606184447 CR: JAGad53655) VxFS directory sticky-bit not functioning according to specification and man pages. Resolution: Make VxFS directory sticky-bit functionality conform to specification and man pages. PHKL_21210: (SR: 5003451625 CR: JAGaa93198) The number of subdirectories which can be created for VxFS is limitted to LINK_MAX. Resolution: A new VxFS tunable (vx_maxlink) has been added which allows this limit to be changed to any value between LINK_MAX and INT_MAX (See limits(5)). PHKL_18796: SR: 1653264663 CR: JAGaa11143 No default was set for quota information, so without a quota file, the user's uid and gid were undefined and invalid. Resolution: Initialize group and user id's properly when there is no quota file and quota is turned on. Enhancement: No (superseded patches contained enhancements) PHKL_28105: Enhancements were delivered in a patch this one has superseded. Please review the Defect Description text for more information. SR: 1653264663 5003436360 5003451625 8606157898 8606184447 8606197982 8606204945 8606233621 8606270211 8606274037 Patch Files: JournalFS.VXFS-BASE-KRN,fr=B.11.00,fa=HP-UX_B.11.00_32,v=HP: /usr/conf/lib/libvxfs_base.a(vx_chain.o) /usr/conf/lib/libvxfs_base.a(vx_config.o) /usr/conf/lib/libvxfs_base.a(vx_dirop.o) /usr/conf/lib/libvxfs_base.a(vx_mount.o) /usr/conf/lib/libvxfs_base.a(vx_vfsops.o) /usr/conf/master.d/vxfs /usr/conf/space.h.d/vxfs.h JournalFS.VXFS-BASE-KRN,fr=B.11.00,fa=HP-UX_B.11.00_64,v=HP: /usr/conf/lib/libvxfs_base.a(vx_chain.o) /usr/conf/lib/libvxfs_base.a(vx_config.o) /usr/conf/lib/libvxfs_base.a(vx_dirop.o) /usr/conf/lib/libvxfs_base.a(vx_mount.o) /usr/conf/lib/libvxfs_base.a(vx_vfsops.o) /usr/conf/master.d/vxfs /usr/conf/space.h.d/vxfs.h what(1) Output: JournalFS.VXFS-BASE-KRN,fr=B.11.00,fa=HP-UX_B.11.00_32,v=HP: /usr/conf/lib/libvxfs_base.a(vx_chain.o): vx_chain.c $Date: 2002/07/16 14:55:39 $Revision: r11 ros/2 PATCH_11.00 (PHKL_27070) /usr/conf/lib/libvxfs_base.a(vx_config.o): vx_config.c $Date: 2002/08/01 09:48:47 $Revision: r1 1ros/7 PATCH_11.00 (PHKL_27648) /usr/conf/lib/libvxfs_base.a(vx_dirop.o): vx_dirop.c $Date: 2001/05/23 11:00:15 $Revision: r11 ros/5 PATCH_11.00 (PHKL_24201) /usr/conf/lib/libvxfs_base.a(vx_mount.o): vx_mount.c $Date: 2002/10/22 23:23:57 $Revision: r11 ros/6 PATCH_11.00 (PHKL_28105) /usr/conf/lib/libvxfs_base.a(vx_vfsops.o): vx_vfsops.c $Date: 2002/08/01 09:47:41 $Revision: r1 1ros/4 PATCH_11.00 (PHKL_27648) /usr/conf/master.d/vxfs: vxfs $Date: 2002/05/22 14:03:33 $Revision: r11ros/3 PATCH_11.00 (PHKL_27070) /usr/conf/space.h.d/vxfs.h: vxfs.h $Date: 2002/06/25 13:03:35 $Revision: r11ros/ 2 PATCH_11.00 (PHKL_27070) JournalFS.VXFS-BASE-KRN,fr=B.11.00,fa=HP-UX_B.11.00_64,v=HP: /usr/conf/lib/libvxfs_base.a(vx_chain.o): vx_chain.c $Date: 2002/07/16 14:55:39 $Revision: r11 ros/2 PATCH_11.00 (PHKL_27070) /usr/conf/lib/libvxfs_base.a(vx_config.o): vx_config.c $Date: 2002/08/01 09:48:47 $Revision: r1 1ros/7 PATCH_11.00 (PHKL_27648) /usr/conf/lib/libvxfs_base.a(vx_dirop.o): vx_dirop.c $Date: 2001/05/23 11:00:15 $Revision: r11 ros/5 PATCH_11.00 (PHKL_24201) /usr/conf/lib/libvxfs_base.a(vx_mount.o): vx_mount.c $Date: 2002/10/22 23:23:57 $Revision: r11 ros/6 PATCH_11.00 (PHKL_28105) /usr/conf/lib/libvxfs_base.a(vx_vfsops.o): vx_vfsops.c $Date: 2002/08/01 09:47:41 $Revision: r1 1ros/4 PATCH_11.00 (PHKL_27648) /usr/conf/master.d/vxfs: vxfs $Date: 2002/05/22 14:09:35 $Revision: r11ros/3 PATCH_11.00 (PHKL_27070) /usr/conf/space.h.d/vxfs.h: vxfs.h $Date: 2002/06/25 13:03:20 $Revision: r11ros/ 2 PATCH_11.00 (PHKL_27070) */ cksum(1) Output: JournalFS.VXFS-BASE-KRN,fr=B.11.00,fa=HP-UX_B.11.00_32,v=HP: 2807601355 6064 /usr/conf/lib/libvxfs_base.a(vx_chain.o) 3855001203 8288 /usr/conf/lib/libvxfs_base.a(vx_config.o) 2959302634 9044 /usr/conf/lib/libvxfs_base.a(vx_dirop.o) 3076481259 32016 /usr/conf/lib/libvxfs_base.a(vx_mount.o) 2692733744 17168 /usr/conf/lib/libvxfs_base.a(vx_vfsops.o) 2323883929 5054 /usr/conf/master.d/vxfs 2229146679 631 /usr/conf/space.h.d/vxfs.h JournalFS.VXFS-BASE-KRN,fr=B.11.00,fa=HP-UX_B.11.00_64,v=HP: 2671912885 11968 /usr/conf/lib/libvxfs_base.a(vx_chain.o) 2593296768 16976 /usr/conf/lib/libvxfs_base.a(vx_config.o) 763713142 15952 /usr/conf/lib/libvxfs_base.a(vx_dirop.o) 261667588 66104 /usr/conf/lib/libvxfs_base.a(vx_mount.o) 998828046 42232 /usr/conf/lib/libvxfs_base.a(vx_vfsops.o) 2823206355 5053 /usr/conf/master.d/vxfs 1990722943 630 /usr/conf/space.h.d/vxfs.h Patch Conflicts: None Patch Dependencies: s700: 11.00: PHKL_18543 s800: 11.00: PHKL_18543 Hardware Dependencies: None Other Dependencies: None Supersedes: PHKL_27648 PHKL_27070 PHKL_26794 PHKL_24905 PHKL_24201 PHKL_22312 PHKL_21210 PHKL_19481 PHKL_18796 Equivalent Patches: None Patch Package Size: 300 KBytes Installation Instructions: Please review all instructions and the Hewlett-Packard SupportLine User Guide or your Hewlett-Packard support terms and conditions for precautions, scope of license, restrictions, and, limitation of liability and warranties, before installing this patch. ------------------------------------------------------------ 1. Back up your system before installing a patch. 2. Login as root. 3. Copy the patch to the /tmp directory. 4. Move to the /tmp directory and unshar the patch: cd /tmp sh PHKL_28105 5. Run swinstall to install the patch: swinstall -x autoreboot=true -x patch_match_target=true \ -s /tmp/PHKL_28105.depot By default swinstall will archive the original software in /var/adm/sw/save/PHKL_28105. If you do not wish to retain a copy of the original software, include the patch_save_files option in the swinstall command above: -x patch_save_files=false WARNING: If patch_save_files is false when a patch is installed, the patch cannot be deinstalled. Please be careful when using this feature. For future reference, the contents of the PHKL_28105.text file is available in the product readme: swlist -l product -a readme -d @ /tmp/PHKL_28105.depot To put this patch on a magnetic tape and install from the tape drive, use the command: dd if=/tmp/PHKL_28105.depot of=/dev/rmt/0m bs=2k Special Installation Instructions: PHKL_27070: A new system-wide tunable, VX_NOTHROTTLE, is introduced to enable or disable I/O throttling in the sync() and write() code paths. By default, I/O throttling will be disabled (VX_NOTHROTTLE is set to 1). I/O throttling is enabled by setting VX_NOTHROTTLE to 0. Most systems should run fine with I/O throttling disabled. However, if long disk I/O queues are preventing other I/O from completing in a timely manner, enabling I/O throttling by setting VX_NOTHROTTLE to 0 may help reduce the disk I/O queues, allowing other I/Os to complete sooner. Please note, enabling I/O throttling may impact flush performance as processes that flush large amounts of data (such as sync(2) and bdf(1M)) will block until the amount of data being flushed decreases. PHKL_21210: After patch is installed the vx_maxlink tunable can be set in /stand/system. At that time the system needs to be rebuilt and rebooted. This patch depends on base patch PHKL_18543. For successful installation please insure that PHKL_18543 is already installed, or that PHKL_18543 is included in the same depot with this patch and PHKL_18543 is selected for installation.