InfoScale 8.0 Update 3 Cumulative Patch on RHEL8 Platform (2024)

更新 ID: UPD826886

版本: 8.0.0.3300

平台: Linux

发布日期: 2024-08-30

 * * * READ ME * * * * * * InfoScale 8.0 * * * * * * Patch 3300 * * * Patch Date: 2024-08-29This document provides the following information: * PATCH NAME * OPERATING SYSTEMS SUPPORTED BY THE PATCH * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * SUMMARY OF INCIDENTS FIXED BY THE PATCH * DETAILS OF INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH * KNOWN ISSUESPATCH NAME----------InfoScale 8.0 Patch 3300OPERATING SYSTEMS SUPPORTED BY THE PATCH----------------------------------------RHEL8 x86-64PACKAGES AFFECTED BY THE PATCH------------------------------VRTSamfVRTSaslapmVRTScavfVRTScpsVRTSdbacVRTSdbedVRTSfsadvVRTSgabVRTSglmVRTSgmsVRTSlltVRTSodmVRTSperlVRTSpythonVRTSrestVRTSsfmhVRTSsptVRTSvcsVRTSvcsagVRTSvcseaVRTSvekiVRTSvxfenVRTSvxfsVRTSvxvmBASE PRODUCT VERSIONS FOR THE PATCH----------------------------------- * InfoScale Availability 8.0 * InfoScale Enterprise 8.0 * InfoScale Foundation 8.0 * InfoScale Storage 8.0SUMMARY OF INCIDENTS FIXED BY THE PATCH---------------------------------------Patch ID: VRTSvxfs-8.0.0.3300* 4097080 (4090032) System might panic in vx_dev_strategy() while Sybase or Oracle configuration.* 4119974 (4119965) VxFS mount binary failed to mount VxFS with SELinux context.* 4129238 (4116887) Running fsck -y on large size metasave with lots of hardlinks is consuming huge amount of system memory.* 4162687 (4145203) Invoking veki through systemctl inside vxfs-startup script.* 4162690 (4134661) Hang seen in the cp command in case of checkpoint promote in cluster filesystem environment.* 4162691 (4084239) Machine hit with Panic because if assert "f:xted_irwunlock:2"* 4162692 (4092440) FSPPADM giving return code 0 (success) despite policy enforcement is failing.* 4162693 (4119961) We hit the assert "xted_irwunlock:2" while doing in-house testing of WORM/Aulog features.* 4162694 (4099740) UX:vxfs mount: ERROR: V-3-21264: <device> is already mounted, <mount-point> is busy, or the allowable number of mount points has been exceeded.* 4162695 (4070819) Handling the case where the error is returned while we are going to get the inode from the last clone which is marked as overlay and is going to be removed.* 4162696 (4058153) FSCK hangs while clearing VX_EXHASH_CLASS attribute in 512 byte FS.* 4162697 (4136235) Includes module parameter for changing pnlct merge frequency.* 4162698 (4134194) vxfs/glm worker thread panic with kernel NULL pointer dereference* 4162699 (4112056) Hitting assert "f:vx_vnode_deinit:1" during in-house FS testing.* 4162700 (4068548) File system log replay fails with "inode marked bad, allocation flags (0x0001)"* 4162701 (4137040) System got hung.* 4162702 (4101075) During in-house CFS testing we hit the assert "vx_extfindbig:4" in extent look path.* 4162703 (4132435) Failures seen in FSQA cmds->fsck tests, panic in get_dotdotlst* 4162704 (4068953) FSCK detected error on 512 byte node FS, in 1 fset ilist while verifying the FS after doing log replay and upgrading the FS to 17 DLV.* 4162706 (4117342) System might panic due to hard lock up detected on CPU* 4162707 (4126943) Create lost+found directory in VxFS file system with default ACL permissions as 700.* 4162708 (4134884) Unable to deport Diskgroup. Volume or plex device is open or attached* 4162709 (4090088) Force unmounting might cause system crash, specifically when "panic_on_warn" is set.* 4162710 (4068201) File system corruption can happen in cases where node which committed the transaction crashed after sending the reply and before flushing to the log.* 4162711 (4149899) Veritas file replication failover fails to perform the failover of job to target cluster.* 4162835 (4142555) Modify reference to running fsck -y in mount.vxfs error message and update fsck_vxfs manpage* 4162853 (4158757) VxFS support for RHEL-8.10.* 4163378 (4075251) Performance bottleneck is seen in internal sequential write when there is very less space available in the filesystem.* 4163379 (4089199) Dynamic reconfiguration operation for CPU takes a lot of time.* 4163383 (4155961) Panic in vx_rwlock during force unmount.* 4163384 (4119281) Higher page-in requests on Solaris 11 SPARC.* 4164534 (4129680) Generate and add changelog in VxFS rpm* 4164961 (4163337) Intermittent df slowness seen across cluster.* 4168371 (4162316) FS migration to VxFS might hit the kernel PANIC if CrowdStrike falcon sensor is running.* 4174512 (4168443) System panicked at vx_clonemap.* 4174514 (4171368) Node panicked while unmounting filesystem.Patch ID: VRTSvxfs-8.0.0.3200* 4096656 (4090032) System might panic in vx_dev_strategy() while Sybase or Oracle configuration.* 4119279 (4119281) Higher page-in requests on Solaris 11 SPARC.* 4126124 (4121691) Unknown messages in /var/log/messages regarding VxFS rules file (/etc/udev/rules.d/60-vxca.rules) for udev* 4136241 (4136110) cmd "umount -l" is unmounting mount points even after adding mntlock in sles12 and sles15.* 4144027 (4126957) System crashes with VxFS stack.* 4144029 (4137040) System got hung.* 4144042 (4112056) Hitting assert "f:vx_vnode_deinit:1" during in-house FS testing.* 4144043 (4126943) Create lost+found directory in VxFS file system with default ACL permissions as 700.* 4144059 (4134661) Hang seen in the cp command in case of checkpoint promote in cluster filesystem environment.* 4144060 (4132435) Failures seen in FSQA cmds->fsck tests, panic in get_dotdotlst* 4144061 (4092440) FSPPADM giving return code 0 (success) despite policy enforcement is failing.* 4144063 (4116887) Running fsck -y on large size metasave with lots of hardlinks is consuming huge amount of system memory.* 4144074 (4117342) System might panic due to hard lock up detected on CPU* 4144082 (4134194) vxfs/glm worker thread panic with kernel NULL pointer dereference* 4144086 (4136235) Includes module parameter for changing pnlct merge frequency.* 4144117 (4099740) UX:vxfs mount: ERROR: V-3-21264: <device> is already mounted, <mount-point> is busy, or the allowable number of mount points has been exceeded.* 4144119 (4134884) Unable to deport Diskgroup. Volume or plex device is open or attached* 4145797 (4145203) Invoking veki through systemctl inside vxfs-startup script.* 4149436 (4141665) Security vulnerabilities exist in the Zlib third-party components used by VxFS.Patch ID: VRTSvxfs-8.0.0.3100* 4154855 (4141665) Security vulnerabilities exist in the Zlib third-party components used by VxFS.Patch ID: VRTSvxfs-8.0.0.2900* 4092518 (4096267) Veritas File Replication jobs might failed when there are large number of jobs run in parallel.* 4097466 (4114176) After failover, job sync fails with error "Device or resource busy".* 4107367 (4108955) VFR job hangs on source if thread creation fails on target.* 4111457 (4117827) For EO compliance, there is a requirement to support 3 types of log file permissions 600, 640 and 644 with 600 being default new eo_perm tunable is added in vxtunefs command to manage the log file permissions.* 4112417 (4094326) mdb invocation displays message "failed to add vx_sl_node_level walker: walk name already in use"* 4118795 (4100021) Running setfacl followed by getfacl resulting in "No such device or address" error.* 4119023 (4116329) While checking FS sanity with the help of "fsck -o full -n" command, we tried to correct the FS flag value (WORM/Softworm), but failed because -n (read-only) option was given.* 4123143 (4123144) fsck binary generating coredumpPatch ID: VRTSvxfs-8.0.0.2700* 4113911 (4113121) VXFS support for RHEL 8.8.* 4114019 (4067505) invalid VX_AF_OVERLAY aflags error in fsck* 4114020 (4083056) Hang observed while punching the smaller hole over the bigger hole.* 4114021 (4101634) Directory inode getting incorrect file-type error in fsck.Patch ID: VRTSvxfs-8.0.0.2600* 4114654 (4114652) VXFS support for RHEL 8.7 minor kernel 4.18.0-425.19.2.Patch ID: VRTSvxfs-8.0.0.2300* 4108381 (4107777) VxFS support for RHEL 8.7 minor kernel.Patch ID: VRTSvxfs-8.0.0.2200* 4100925 (4100926) VxFS module failed to load on RHEL8.7Patch ID: VRTSvxfs-8.0.0.2100* 4095889 (4095888) Security vulnerabilities exist in the Sqlite third-party components used by VxFS.Patch ID: VRTSvxfs-8.0.0.1800* 4068960 (4073203) Veritas file replication might generate a core while replicating the files to target.* 4071108 (3988752) Use ldi_strategy() routine instead of bdev_strategy() for IO's in solaris.* 4072228 (4037035) VxFS should have the ability to control the number of inactive processing threads.* 4078335 (4076412) Addressing Executive Order (EO) 14028, initial requirements which is intended to improve the Federal Governments investigative and remediation capabilities related to cybersecurity incidents.* 4078520 (4058444) Loop mounts using files on VxFS fail on Linux systems.* 4079142 (4077766) VxFS kernel module might leak memory during readahead of directory blocks.* 4079173 (4070217) Command fsck might fail with 'cluster reservation failed for volume' message for a disabled cluster-mounted filesystem.* 4082260 (4070814) Security Vulnerability observed in Zlib a third party component VxFS uses.* 4082865 (4079622) Existing migration read/write iter operation handling is not fully functional as vxfs uses normal read/write file operation only.* 4083335 (4076098) Fix migration issues seen with falcon-sensor.* 4085623 (4085624) While running fsck, fsck might dump core.* 4085839 (4085838) Command fsck may generate core due to processing of zero size attribute inode.* 4086085 (4086084) VxFS mount operation causes system panic.* 4088341 (4065575) Write operation might be unresponsive on a local mounted VxFS filesystem in a no-space conditionPatch ID: VRTSvxfs-8.0.0.1700* 4081150 (4079869) Security Vulnerability in VxFS third party components* 4083948 (4070814) Security Vulnerability in VxFS third party component ZlibPatch ID: VRTSvxfs-8.0.0.1200* 4055808 (4062971) Enable partition directory on WORM file system* 4056684 (4056682) New features information on a filesystem with fsadm(file system administration utility) from a device is not displayed.* 4062606 (4062605) Minimum retention time cannot be set if the maximum retention time is not set.* 4065565 (4065669) Creating non-WORM checkpoints fails when the tunables - minimum retention time and maximum retention time are set.* 4065651 (4065666) Enable partition directory on WORM file system having WORM enabled on files with retention period not expired.Patch ID: VRTSvxfs-8.0.0.1100* 4061114 (4052883) VxFS support for RHEL 8.5.Patch ID: VRTSdbed-8.0.0.1800* 4079372 (4073842) SFAE changes to support Oracle 21cPatch ID: VRTSdbac-8.0.0.2200* 4108954 (4107779) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).Patch ID: VRTSdbac-8.0.0.2100* 4100204 (4100203) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).Patch ID: VRTSdbac-8.0.0.1900* 4090090 (4090485) Installation of Oracle 12c GRID and database fails on RHEL8.*/OL8.* with GLIBC package errorPatch ID: VRTSdbac-8.0.0.1800* 4089728 (4089722) VRTSgab , VRTSamf and VRTSdbed driver does not load on RHEL and SLES platform.Patch ID: VRTSdbac-8.0.0.1100* 4053178 (4053171) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).Patch ID: VRTSveki-8.0.0.2500* 4118568 (4110457) Veki packaging were failing due to dependencyPatch ID: VRTSveki-8.0.0.1800* 4056647 (4055072) Upgrading VRTSveki package using yum reports errorPatch ID: VRTSveki-8.0.0.1200* 4070027 (4066550) Increasing Veki systemd service start timeout to 300 secondsPatch ID: VRTSfsadv-8.0.0.2200* 4103001 (4103002) Replication failures observed in internal testingPatch ID: VRTSfsadv-8.0.0.2100* 4092150 (4088024) Security vulnerabilities exist in the OpenSSL third-party components used by VxFS.Patch ID: VRTSrest-2.0.0.1300* 4088973 (4089451) When a read-only filesystem was created on a Volume, GET on mountpoint's details was throwing error .* 4089033 (4089453) Some VCS REST APIs were crashing the Gunicorn worker.* 4089041 (4089449) GET resources API on empty service group was throwing an error.* 4089046 (4089448) Logging in REST API is not EO-compliant.Patch ID: -3.9.2.24* 4114375 (4113851) For VRTSpython need to fix some open CVE'sPatch ID: -5.34.0.4* 4072234 (4069607) Security vulnerability detected on VRTSperl 5.34.0.0 released with Infoscale 8.0.* 4075150 (4075149) Security vulnerabilities detected in OpenSSL packaged with VRTSperl/VRTSpython released with Infoscale 8.0.Patch ID: VRTSsfmh-HF0800510* 4157270 (4157265) sfmh for IS 8.0 RHEL8.9 Platform PatchPatch ID: -8.0.0.1700* 4117339 (4132683) Need a tool to collect IS product specific information from both standalone node and clustered env.* 4124177 (4125116) Logs collected from different tools of VRTSspt are stored at different location.* 4129889 (4114988) No Progress status reporting through long running metasave utility* 4139975 (4149462) New script is provided list_missing_incidents.py which compares changelogs of rpm and lists missing incidents in new version.* 4146957 (4149448) New script is provided check_incident_inchangelog.py which will check if incident abstract is present in changelog.Patch ID: VRTSvcs-8.0.0.2700* 4161822 (4129493) Tenable security scan kills the Notifier resource.* 4162953 (4136359) When upgrading InfoScale with latest Public Patch Bundle or VRTSvcsag package, types.cf is updated.Patch ID: VRTSvcs-8.0.0.2400* 4111623 (4100720) GCO fails to configure for the latest RHEL/SLES platforms.Patch ID: VRTSvcs-8.0.0.2100* 4103077 (4103073) Upgrading Netsnmp component to fix security vulnerabilities .Patch ID: VRTSvcs-8.0.0.1800* 4084675 (4089059) gcoconfig.log file permission is changes to 0600.Patch ID: VRTSvcs-8.0.0.1400* 4065820 (4065819) Protocol version upgrade failed.Patch ID: VRTSvxfen-8.0.0.2700* 4114265 (4122405) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).* 4117657 (4108561) Reading vxfen reservation not working* 4173687 (4166666) Failed to configure Disk based fencing on rdm mapped devices from KVM host to kvm guest* 4176817 (4176110) vxfentsthdw failed to verify fencing disks compatibility in KVM environment* 4177719 (4113847) Support for even number of coordination disks for CVM-based disk-based fencingPatch ID: VRTSvxfen-8.0.0.2200* 4108953 (4107779) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).Patch ID: VRTSvxfen-8.0.0.2100* 4102352 (4100203) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).Patch ID: VRTSvxfen-8.0.0.1800* 4087166 (4087134) The error message 'Touch /var/VRTSvcs/log/vxfen/vxfen.log failed' appears after starting vxfen service.* 4088061 (4089052) On RHEL9, Coordination Point Replacement operation is causing node panicPatch ID: VRTSvxfen-8.0.0.1500* 4072717 (4072335) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 6(RHEL8.6).Patch ID: VRTSvxfen-8.0.0.1200* 3951882 (4004248) vxfend generates core sometimes during vxfen race in CPS based fencing configurationPatch ID: VRTSvxfen-8.0.0.1100* 4064785 (4053171) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).Patch ID: VRTSgab-8.0.0.2700* 4113341 (4113340) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 8(RHEL8.8).Patch ID: VRTSgab-8.0.0.2200* 4108951 (4107779) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).Patch ID: VRTSgab-8.0.0.2100* 4100453 (4100203) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).Patch ID: VRTSgab-8.0.0.1800* 4089723 (4089722) VRTSgab , VRTSamf and VRTSdbed driver does not load on RHEL and SLES platform.Patch ID: VRTSgab-8.0.0.1500* 4073696 (4072335) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 6(RHEL8.6).Patch ID: VRTSgab-8.0.0.1100* 4064784 (4053171) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).Patch ID: VRTSllt-8.0.0.2700* 4135413 (4084657) RHEL8/7.4.1 new installation, fencing/LLT panic while using TCP over LLT.* 4135420 (3989372) When the CPU load and memory consumption is high in a VMware environment, some nodes in an InfoScale cluster may get fenced out.* 4136152 (4124759) Panic happened with llt_ioship_recv on a server running in AWS.* 4145248 (4139781) Unexpected or corrupted skb, memory type missing in buffer header.* 4156794 (4135825) Once root file system is full during llt start, llt module failing to load forever.* 4156815 (4087543) Node panic observed at llt_rdma_process_ack+189Patch ID: VRTSllt-8.0.0.2600* 4135413 (4084657) RHEL8/7.4.1 new installation, fencing/LLT panic while using TCP over LLT.* 4135420 (3989372) When the CPU load and memory consumption is high in a VMware environment, some nodes in an InfoScale cluster may get fenced out.* 4136152 (4124759) Panic happened with llt_ioship_recv on a server running in AWS.* 4145248 (4139781) Unexpected or corrupted skb, memory type missing in buffer header.* 4156794 (4135825) Once root file system is full during llt start, llt module failing to load forever.* 4156815 (4087543) Node panic observed at llt_rdma_process_ack+189* 4156824 (4138779) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 9(RHEL8.9).Patch ID: VRTSllt-8.0.0.2400* 4116421 (4113340) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 8(RHEL8.8).Patch ID: VRTSllt-8.0.0.2200* 4108947 (4107779) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).Patch ID: VRTSllt-8.0.0.2100* 4101232 (4100203) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).Patch ID: VRTSllt-8.0.0.1800* 4061158 (4061156) IO error on /sys/kernel/slab folder* 4079637 (4079636) LLT over IPsec is causing node crash* 4079662 (3981917) Support LLT UDP multiport on 1500 MTU based networks.* 4080630 (4046953) Delete / disable confusing messages.Patch ID: VRTSllt-8.0.0.1500* 4073695 (4072335) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 6(RHEL8.6).Patch ID: VRTSllt-8.0.0.1200* 4066063 (4066062) Node panic is observed while using llt udp with multiport enabled.* 4066667 (4040261) During LLT configuration, if set-verbose is set to 1 in /etc/llttab, an lltconfig core dump is observed.Patch ID: VRTSllt-8.0.0.1100* 4064783 (4053171) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).Patch ID: VRTSvcsea-8.0.0.2500* 4118769 (4073508) Oracle virtual fire-drill is failing.Patch ID: VRTSvcsea-8.0.0.1800* 4030767 (4088595) hapdbmigrate utility fails to online the oracle service group* 4079559 (4064917) As a part of Oracle 21c support, fixed the issue Oracle agent fails to generate ora_api using the build_oraapi.sh scriptPatch ID: VRTSvcsag-8.0.0.2700* 4121275 (4121270) EBSvol agent error in attach disk : RHEL 7.9 + Infoscale 8.0 on AWS instance type c6i.large with NVME devices.* 4122004 (4122001) NIC resource remain online after unplug network cable on ESXi server.* 4127323 (4127320) The ProcessOnOnly agent fails to bring online a resource when a user shell is set to /sbin/nologin.* 4161344 (4152812) AWS EBSVol agent takes long time to perform online and offline operations on resources.* 4161350 (4152815) AWS EBS Volume in-use with other AWS instance is getting used by cluster nodes through AWS EBSVol agent.* 4162952 (4142040) While upgrading the VRTSvcsag rpm package, the '/etc/VRTSvcs.conf/config/types.cf' file on Veritas Cluster Server(VCS) might be incorrectly updated.* 4163231 (4152700) When Private DNS Zone resource ID is passed, the AzureDNSZone Agent returns an error saying that the resource cannot be found.* 4163234 (4152886) AWSIP agent fails to bring OverlayIP resources online and offline on the instances in a shared VPC.* 4165268 (4162658) LVMVolumeGroup resource fails to offline/clean in cloud environment after path failure.* 4169950 (4169794) Add a new attribute 'LinkTest' to the NIC agent to monitor a n/w device for which no IP is configuredPatch ID: VRTSvcsag-8.0.0.2500* 4118318 (4113151) VMwareDisksAgent reports resource online before VMware disk to be online is present into vxvm/dmp database.* 4118448 (4075950) IPv6 neighbor flush logic needs to be added to IP/MultiNIC agents* 4118455 (4118454) Process agent fails to come online when root user shell is set to /sbin/nologin.* 4118767 (4094539) Agent resource monitor not parsing process name correctly.Patch ID: VRTSvcsag-8.0.0.2400* 4113057 (4113056) ReuseMntPt is not honored when the same mountpoint is used for two resources with different FSType.Patch ID: VRTSvcsag-8.0.0.1800* 4030767 (4088595) hapdbmigrate utility fails to online the oracle service group* 4058802 (4073842) SFAE changes to support Oracle 21c* 4079372 (4073842) SFAE changes to support Oracle 21c* 4079559 (4064917) As a part of Oracle 21c support, fixed the issue Oracle agent fails to generate ora_api using the build_oraapi.sh script* 4081774 (4083099) AzureIP resource fails to go offline when OverlayIP is configured.Patch ID: VRTScps-8.0.0.2700* 4152884 (4152882) vxcpserv process received SIGABRT signal due to invalid pointer access in acvsc_lib while writing logs.Patch ID: VRTScps-8.0.0.1900* 4091306 (4088158) Security vulnerabilities exists in Sqlite third-party components used by VCS.Patch ID: VRTScps-8.0.0.1800* 4073050 (4018218) Secure communication between a CP Server and a CP Client cannot be established using TLSv1.2Patch ID: VRTScps-8.0.0.1200* 4066225 (4056666) The Error writing to database message may intermittently appear in syslogs on CP servers.Patch ID: VRTSamf-8.0.0.2700* 4145249 (4136003) A cluster node panics when the AMF module overruns internal buffer to analyze arguments of an executable binary.* 4162992 (4161644) System panics when AMF enabled and there are Process/Application resources.Patch ID: VRTSamf-8.0.0.2400* 4116411 (4113340) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 8(RHEL8.8).Patch ID: VRTSamf-8.0.0.2200* 4108952 (4107779) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).Patch ID: VRTSamf-8.0.0.2100* 4101233 (4100203) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).Patch ID: VRTSamf-8.0.0.1800* 4089724 (4089722) VRTSgab , VRTSamf and VRTSdbed driver does not load on RHEL and SLES platform.Patch ID: VRTSamf-8.0.0.1500* 4072752 (4072335) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 6(RHEL8.6).Patch ID: VRTSamf-8.0.0.1100* 4064788 (4053171) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).Patch ID: VRTSvxvm-8.0.0.2900* 4058590 (4058867) VxVM rpm Support on RHEL 8.7 minor kernel 4.18.0-425.10.1.el8_7.x86_64* 4067237 (4058894) Messages in /var/log/messages regarding "ignore_device".* 4070100 (3159650) Implemented vol_vvr_use_nat tunable support for vxtune.* 4103795 (4011780) Add support for DELL EMC PowerStore plus PP* 4104264 (4098391) Continuous system crash is observed during VxVM installation.* 4104757 (4055159) vxdisk list showing incorrect value of LUN_SIZE for nvme disks* 4104984 (4095163) system panic due to a race freeing VVR update.* 4105598 (4107801) /dev/vx/.dmp hardware path entries are not getting created on SLES15SP3 onwards.* 4108322 (4107083) In case of EMC BCV NR LUNs, vxconfigd taking a long time to start post reboot.* 4108392 (4107802) Fix for calculating best-fit module for upcoming RHEL8.7 minor kernels (higher than 4.18.0-425.10.1.el8_7.x86_64).* 4109554 (4105953) system panic due to VVR accessed a NULL pointer.* 4110398 (4106254) Nodes crashed in shared-nothing (Flexible Shared Storage) environment if node reboot followed by NVME disk failure is executed* 4111009 (4108475) vxfentsthdw script failed with "Expect no writes for disks ... "* 4111309 (4091390) vradmind service has dump core and stopped on few nodes* 4111437 (4086131) Vxrlink dumped core when rlink is upgraded as a part of dg upgrade.* 4111442 (4066785) create new option usereplicatedev=only to import the replicated LUN only.* 4111560 (4098391) Continuous system crash is observed during VxVM installation.* 4112219 (4069134) "vxassist maxsize alloc:array:<enclosure_name>" command may fail.* 4112549 (4112701) Nodes stuck in reconfig hang and vxconfigd coredump after rebooting all nodes with a delay of 5min in between them.* 4112606 (4100069) One of standard disk groups fails to auto-import with 'Disk for disk group not found' error when those disk groups co-exist with the cloned disk group.* 4112610 (4102532) /etc/default/vxsf file gets world write permission when "vxtune storage_connectivity asymmetric" is run.* 4112884 (4107401) Replication stopped after VVR logowner reboot* 4113223 (4093067) System panic occurs because of NULL pointer in block device structure.* 4113225 (4068090) System panic occurs because of block device inode ref count leaks.* 4113310 (4114601) Panic: in dmp_process_errbp() for disk pull scenario.* 4113328 (4102439) Volume Manager Encryption EKM Key Rotation (vxencrypt rekey) Operation Fails on IS 7.4.2/rhel7* 4113331 (4105565) In CVR environment, system panic due to NULL pointer when VVR was doing recovery.* 4113342 (4098965) Crash at memset function due to invalid memory access.* 4113357 (4112433) Security vulnerabilities exists in third party components [openssl, curl and libxml].* 4113691 (4064772) After enabling slub debug, system could hang with IO load.* 4113692 (4089626) Create XFS on VxDMP devices hang as VxDMP doesn't support chained BIO.* 4113693 (4100775) vxconfigd was hung as VxDMP doesn't support chained BIO on rhel7.* 4113694 (4113323) VxVM Support on RHEL 8.8* 4114963 (4114962) [NBFS-3.1][DL]:MASTER and CAT_FS got corrupted while performing multiple NVMEs failure* 4115251 (4115195) [NBFS-3.1][DL]:MEDIA_FS got corrupted after panic loop test* 4115252 (4115193) Data corruption observed after the node fault and cluster restart in DR environment* 4115381 (4091783) build script and mb.sh changes in unixvm for integration of storageapi* 4116548 (4111254) vradmind dumps core while associating a rlink to rvg because of NULL pointer reference.* 4116551 (4108913) Vradmind dumps core because of memory corruption.* 4116557 (4085404) Huge perf drop after Veritas Volume Replicator (VVR) entered Data Change Map (DCM) mode, when a large size of Storage Replicator Log (SRL) is configured.* 4116559 (4091076) SRL gets into pass-thru mode because of head error.* 4116562 (4114257) Observed IO hung and high system load average after rebooted master and one slave node rejoins cluster.* 4116565 (4034741) The current fix from limits IO load on secondary causing deadlock situtaion* 4116567 (4072862) Stop cluster hang because of RVGLogowner and CVMClus resources fail to offline.* 4116688 (4085145) System with NVME devices can crash due to memory corruption.* 4117110 (4113841) VVR panic in replication connection handshake request from network scan tool.* 4117385 (4117350) Import operation on disk group created on Hitachi ShadowImage (SI) disks is failing .* 4118108 (4114867) systemd-udevd[2224]: invalid key/value pair in file /etc/udev/rules.d/41-VxVM-selinux.rules on line 20, starting at character 103 ('D')* 4118111 (4065490) VxVM udev rules consumes more CPU and appears in "top" output when system has thousands of storage devices attached.* 4118845 (4116024) machine panic due to access illegal address.* 4119257 (4090772) vxconfigd/vx commands hung if fdisk opened secondary volume and secondary logowner panic'd* 4119276 (4090943) VVR Primary RLink cannot connect as secondary reports SRL log is full.* 4119438 (4117985) EC volume corruption due to lockless access of FPU* 4119553 (4118510) Volume manager tunable to control log file permissions* 4119852 (4081740) vxdg flush command slow due to too many luns needlessly access /proc/partitions.* 4120194 (4120191) IO hang occurred when flushing SRL to DCM because of a deadlock issue.* 4120350 (4120878) After enabling the dmp_native_support, system failed to boot.* 4124887 (4090828) Enhancement to track plex att/recovery data synced in past to debug corruption* 4126293 (4114927) Failed to mount /boot on dmp device after enabling dmp_native_support.* 4127273 (4123600) Panic generated in kms encryption testing* 4127934 (4124223) Core dump is generated for vxconfigd in TC execution.* 4128248 (4105204) Node not able to join the cluster after iLO "press and hold" scenario in loop* 4136205 (4111978) Replication failed to start due to vxnetd threads not running on secondary site.* 4136892 (4136458) In CVR environment, the DCM resync may hang with 0% sync remaining.* 4140218 (4117568) vradmind dumps core due to invalid memory access.* 4149249 (4149248) Security vulnerabilities have been discovered in third-party components (OpenSSL, Curl, and libxml) employed by VxVM.* 4162732 (4073653) VxVM commands get hung after pause-resume and resync operation in CVR setup.* 4162734 (4098144) vxtask list shows the parent process without any sub-tasks which never progresses for SRL volume* 4162735 (4132265) Machine attached with NVMe devices may panic.* 4162741 (4128351) System hung observed when switching log owner.* 4162742 (4122061) Observing hung after resync operation, vxconfigd was waiting for slaves' response* 4162743 (4087628) CVM goes into faulted state when slave node of primary is rebooted .* 4162745 (4145063) unknown symbol message logged in syslogs while inserting vxio module.* 4162747 (4152014) the excluded dmpnodes are visible after system reboot when SELinux is disabled.* 4162748 (4132799) No detailed error messages while joining CVM fail.* 4162749 (4134790) Hardware Replicated DG was marked with clone flag on SLAVEs.* 4162750 (4077944) In VVR environment, application I/O operation may get hung.* 4162751 (4132221) Supportability requirement for easier path link to dmpdr utility* 4162754 (4154121) add a new tunable use_hw_replicatedev to enable Volume Manager to import the hardware replicated disk group.* 4162756 (4159403) add clearclone option automatically when import the hardware replicated disk group.* 4162757 (4160883) clone_flag was set on srdf-r1 disks after reboot.* 4163989 (4162873) disk reclaim is slow.* 4164137 (3972344) vxrecover returns an error - 'ERROR V-5-1-11150' Volume <vol_name> not found'* 4164248 (4162349) vxstat not showing data under MIN and MAX header when using -S option* 4164276 (4142772) Error mask NM_ERR_DCM_ACTIVE on rlink may not be cleared resulting in the rlink being unable to get into DCM again.* 4164312 (4133793) vxsnap restore failed with DCO IO errors during the operation when run in loop for multiple VxVM volumes.* 4164693 (4149498) Getting unsupported .ko files not found warning while upgrading VM packages.* 4164698 (4141806) Write_pattern got hung on primary master.* 4164704 (4136974) IPv6: With multiple RVGs, restarting vxnetd results in rlink disconnect* 4167050 (4134069) VVR replication was not using VxFS SmartMove feature if filesystem was not mounted on RVG Logowner node.* 4167712 (4166086) Invalid device symbolic links exist under root.* 4167866 (4165971) /var/tmp/rpm-tmp.Kl3ycu: line 657: [: missing `]'* 4175213 (4153457) When using Dell/EMC PowerFlex ScaleIO storage, Veritas File System(VxFS) on Veritas Volume Manager(VxVM) volumes fail to mount after reboot.Patch ID: VRTSaslapm 8.0.0.2900* 4177622 (4177623) RHEL8.10 supportPatch ID: VRTSvxvm-8.0.0.2800* 4093140 (4093067) System panic occurs because of NULL pointer in block device structure.* 4130643 (4130642) node failed to rejoin the cluster after this node switched from master to slave due to the failure of the replicated diskgroup import.* 4132798 (4132799) No detailed error messages while joining CVM fail.* 4134024 (4134023) vxconfigrestore(Diskgroup configuration restoration) for H/W Replicated diskgroup failed.* 4134791 (4134790) Hardware Replicated DG was marked with clone flag on SLAVEs.* 4146456 (4128351) System hung observed when switching log owner.* 4146458 (4122061) Observing hung after resync operation, vxconfigd was waiting for slaves' response* 4146462 (4087628) CVM goes into faulted state when slave node of primary is rebooted .* 4146472 (4115078) vxconfigd hung was observed when reboot all nodes of the primary site.* 4149249 (4149248) Security vulnerabilities have been discovered in third-party components (OpenSSL, Curl, and libxml) employed by VxVM.* 4149423 (4145063) unknown symbol message logged in syslogs while inserting vxio module.* 4156836 (4152014) the excluded dmpnodes are visible after system reboot when SELinux is disabled.* 4156837 (4134790) Hardware Replicated DG was marked with clone flag on SLAVEs.* 4156839 (4077944) In VVR environment, application I/O operation may get hung.* 4156841 (4142054) primary master got panicked with ted assert during the run.Patch ID: VRTSaslapm 8.0.0.2800* 4158230 (4158234) ASLAPM rpm Support on RHEL 8.9Patch ID: VRTSvxvm-8.0.0.2700* 4154821 (4149248) Security vulnerabilities have been discovered in third-party components (OpenSSL, Curl, and libxml) employed by VxVM.Patch ID: VRTSvxvm-8.0.0.2600* 4067237 (4058894) Messages in /var/log/messages regarding "ignore_device".* 4109554 (4105953) system panic due to VVR accessed a NULL pointer.* 4111442 (4066785) create new option usereplicatedev=only to import the replicated LUN only.* 4112549 (4112701) Nodes stuck in reconfig hang and vxconfigd coredump after rebooting all nodes with a delay of 5min in between them.* 4113310 (4114601) Panic: in dmp_process_errbp() for disk pull scenario.* 4113357 (4112433) Security vulnerabilities exists in third party components [openssl, curl and libxml].* 4114963 (4114962) [NBFS-3.1][DL]:MASTER and CAT_FS got corrupted while performing multiple NVMEs failure* 4115251 (4115195) [NBFS-3.1][DL]:MEDIA_FS got corrupted after panic loop test* 4115252 (4115193) Data corruption observed after the node fault and cluster restart in DR environment* 4115381 (4091783) build script and mb.sh changes in unixvm for integration of storageapi* 4116548 (4111254) vradmind dumps core while associating a rlink to rvg because of NULL pointer reference.* 4116551 (4108913) Vradmind dumps core because of memory corruption.* 4116557 (4085404) Huge perf drop after Veritas Volume Replicator (VVR) entered Data Change Map (DCM) mode, when a large size of Storage Replicator Log (SRL) is configured.* 4116559 (4091076) SRL gets into pass-thru mode because of head error.* 4116562 (4114257) Observed IO hung and high system load average after rebooted master and one slave node rejoins cluster.* 4116565 (4034741) The current fix from limits IO load on secondary causing deadlock situtaion* 4116567 (4072862) Stop cluster hang because of RVGLogowner and CVMClus resources fail to offline.* 4117110 (4113841) VVR panic in replication connection handshake request from network scan tool.* 4118108 (4114867) systemd-udevd[2224]: invalid key/value pair in file /etc/udev/rules.d/41-VxVM-selinux.rules on line 20, starting at character 103 ('D')* 4118111 (4065490) VxVM udev rules consumes more CPU and appears in "top" output when system has thousands of storage devices attached.* 4118733 (4106689) Solaris Zones cannot be started due to Method "/lib/svc/method/fs-local" failed with exit status 95* 4118845 (4116024) machine panic due to access illegal address.* 4119087 (4067191) IS8.0_SUSE15_CVR: After rebooted slave node master node got panic* 4119257 (4090772) vxconfigd/vx commands hung if fdisk opened secondary volume and secondary logowner panic'd* 4119276 (4090943) VVR Primary RLink cannot connect as secondary reports SRL log is full.* 4119438 (4117985) EC volume corruption due to lockless access of FPU* 4120350 (4120878) After enabling the dmp_native_support, system failed to boot.* 4121241 (4114927) Failed to mount /boot on dmp device after enabling dmp_native_support.* 4121714 (4081740) vxdg flush command slow due to too many luns needlessly access /proc/partitions.Patch ID: VRTSvxvm-8.0.0.2400* 4110560 (4104927) Changing the attributes in vxvm-boot.service for SLES15 is causing regression in RHEL versions.* 4113324 (4113323) VxVM Support on RHEL 8.8* 4113661 (4091076) SRL gets into pass-thru mode because of head error.* 4113663 (4095163) system panic due to a race freeing VVR update.* 4113664 (4091390) vradmind service has dump core and stopped on few nodes* 4113666 (4064772) After enabling slub debug, system could hang with IO load.Patch ID: VRTSvxvm-8.0.0.2200* 4058590 (4058867) VxVM rpm Support on RHEL 8.7 minor kernel 4.18.0-425.10.1.el8_7.x86_64* 4108392 (4107802) Fix for calculating best-fit module for upcoming RHEL8.7 minor kernels (higher than 4.18.0-425.10.1.el8_7.x86_64).Patch ID: VRTSaslapm-8.0.0.2200* 4108933 (4107932) ASLAPM rpm Support on RHEL 8.7 minor kernel 4.18.0-425.10.1.el8_7.x86_64Patch ID: VRTSaslapm-8.0.0.2100* 4102502 (4102501) A security vulnerability exists in the third-party component libcurl.Patch ID: -8.0.0.1900* 4102924 (4101128) VxVM rpm Support on RHEL 8.7 kernelPatch ID: VRTSaslapm-8.0.0.1900* 4102973 (4101139) ASLAPM rpm Support on RHEL 8.7 kernelPatch ID: -8.0.0.1800* 4067609 (4058464) vradmin resizevol fails when FS is not mounted on master.* 4067635 (4059982) vradmind need not check for rlink connect during migrate.* 4070098 (4071345) Unplanned fallback synchronisation is unresponsive* 4078531 (4075860) Tutil putil rebalance flag is not getting cleared during +4 or more node addition* 4079345 (4069940) FS mount failed during Cluster configuration on 24-node physical HP BOM2 setup.* 4080041 (4056953) 3PAR PE LUNs are reported in error state by 3PAR ASL.* 4080105 (4045837) Sub disks are in relocate state after exceed fault slave node panic.* 4080122 (4044068) After disc replacement, Replace Node operation failed at Configure Netbackup stage.* 4080269 (4044898) Copy rlink tags from reprec to info rec, through vxdg upgrade path.* 4080276 (4065145) multivolume and vset not able to overwrite encryption tags on secondary.* 4080277 (3966157) SRL batching feature is broken* 4080579 (4077876) System is crashed when EC log replay is in progress after node reboot.* 4080845 (4058166) Increase DCM log size based on volume size without exceeding region size limit of 4mb.* 4080846 (4058437) Replication between 8.0 and 7.4.x fails due to sector size field.* 4081790 (4080373) SFCFSHA configuration failed on RHEL 8.4.* 4083337 (4081890) On RHEL8 NBFS/Access commands like python3, sort, sudo, ssh, etc are generating core dump during execution of the command mkfs.vxfs mkfs.ext4.* 4085619 (4086718) VxVM modules fail to load with latest minor kernel of SLES15SP2* 4087233 (4086856) For Appliance FLEX product using VRTSdocker-plugin, need to add platform-specific dependencies service ( docker.service and podman.service ) change.* 4087439 (4088934) Kernel Panic while running LM/CFS CONFORMANCE - variant (SLES15SP3)* 4087791 (4087770) NBFS: Data corruption due to skipped full-resync of detached mirrors of volume after DCO repair operation* 4088076 (4054685) In case of CVR environment, RVG recovery gets hung in linux platforms.* 4088483 (4088484) Failed to load DMP_APM NVME modules* 4088762 (4087099) DG is not imported after upgrade to InfoScale 8.0u1 on RHEL8.6.Patch ID: VRTSaslapm-8.0.0.1800* 4080041 (4056953) 3PAR PE LUNs are reported in error state by 3PAR ASL.* 4088762 (4087099) DG is not imported after upgrade to InfoScale 8.0u1 on RHEL8.6.* 4081684 (4082799) A security vulnerability exists in the third-party component libcurl.Patch ID: -8.0.0.1600* 4057420 (4060462) Nidmap information is not cleared after a node leaves, resulting in add node failure subsequently.* 4062799 (4064208) Node failed to join the existing cluster after bits are upgraded to a newer version.* 4065841 (4065495) Add support for DELL EMC PowerStore.* 4066213 (4052580) Supporting multipathing for NVMe devices under VxVM.* 4068407 (4068404) ASL request for HPE 3PAR/Primera/Alletra 9000 ALUA support.Patch ID: VRTSaslapm-8.0.0.1600* 4065841 (4065495) Add support for DELL EMC PowerStore.* 4068407 (4068404) ASL request for HPE 3PAR/Primera/Alletra 9000 ALUA support.Patch ID: -8.0.0.1200* 4066259 (4062576) hastop -local never finishes on Rhel8.4 and RHEL8.5 servers with latest minor kernels due to hang in vxdg deport command.Patch ID: -8.0.0.1100* 4064786 (4053230) VxVM support for RHEL 8.5* 4065628 (4065627) VxVM Modules failed to load after OS upgrade .Patch ID: VRTScavf-8.0.0.3300* 4162960 (4153873) Deport decision was being dependent on local system only not on all systems in the clusterPatch ID: VRTScavf-8.0.0.2800* 4118779 (4074274) DR test and failover activity might not succeed for hardware-replicated disk groups.Patch ID: VRTSgms-8.0.0.3300* 4126370 (4129707) Generate and add changelog in GMS rpmPatch ID: VRTSgms-8.0.0.2800* 4057427 (4057176) Rebooting the system results into emergency mode due to corruption of module dependency files.Patch ID: VRTSgms-8.0.0.2300* 4108584 (4107753) GMS support for RHEL 8.7 minor kernel.Patch ID: VRTSgms-8.0.0.2200* 4101000 (4100999) GMS support for RHEL 8.7.Patch ID: VRTSglm-8.0.0.3300* 4162713 (4126298) System may panic due to unable to handle kernel paging request and memory corruption could happen.Patch ID: VRTSglm-8.0.0.2300* 4108582 (4107754) GLM support for RHEL 8.7 minor kernel.Patch ID: VRTSglm-8.0.0.2200* 4100995 (4100994) GLM module failed to load on RHEL8.7Patch ID: VRTSodm-8.0.0.3300* 4162852 (4159291) ODM support for RHEL-8.10.* 4164549 (4129837) Generate and add changelog in ODM rpmPatch ID: VRTSodm-8.0.0.3200* 4144128 (4126256) no symbol version warning for VEKI's symbol in dmesg after SFCFSHA configuration* 4144301 (4118154) System may panic in simple_unlock_mem() when errcheckdetail enabled.Patch ID: VRTSodm-8.0.0.3100* 4154894 (4144269) After installing VRTSvxfs ODM fails to start.Patch ID: VRTSodm-8.0.0.2900* 4057432 (4056673) Rebooting the system results into emergency mode due to corruption of module dependency files. Incorrect vxgms dependency in odm service file.Patch ID: VRTSodm-8.0.0.2700* 4113912 (4113118) ODM support for RHEL 8.8.Patch ID: VRTSodm-8.0.0.2600* 4114656 (4114655) ODM support for RHEL 8.7 minor kernel 4.18.0-425.19.2.Patch ID: VRTSodm-8.0.0.2300* 4108585 (4107778) ODM support for RHEL 8.7 minor kernel.Patch ID: VRTSodm-8.0.0.2200* 4100923 (4100922) ODM module failed to load on RHEL8.7DETAILS OF INCIDENTS FIXED BY THE PATCH---------------------------------------This patch fixes the following incidents:Patch ID: VRTSvxfs-8.0.0.3300* 4097080 (Tracking ID: 4090032)SYMPTOM:System might panic in vx_dev_strategy() while Sybase or Oracle configuration, the panic stack looks like following:vx_dev_strategyvx_dio_physiovx_dio_rdwrivx_write_directvx_write1vx_write_common_slowvx_write_commonvx_writefop_writepwriteDESCRIPTION:When we are allocating a different buffer, vx_dev_strategy() unable to find the LDI handle.RESOLUTION:Code is modified to fix this issue.* 4119974 (Tracking ID: 4119965)SYMPTOM:VxFS mount binary failed to mount VxFS with SELinux context.DESCRIPTION:Mounting the file system using VxFS binary with specific SELinux context shows below error:/FSQA/fsqa/vxfsbin/mount -t vxfs /dev/vx/dsk/testdg/vol1 /mnt1 -ocontext="system_u:object_r:httpd_sys_content_t:s0"UX:vxfs mount: ERROR: V-3-28681: Selinux context is invalid or option/operation is not supported. Please look into the syslog for more information.RESOLUTION:VxFS mount command is modified to pass context options to kernel only if SELinux is enabled.* 4129238 (Tracking ID: 4116887)SYMPTOM:Running fsck -y on large size metasave with lots of hardlinks is consuming huge amount of system memory.DESCRIPTION:On a FS with lot of hardlinks, requires a lot of memory for storing dotdot information in memory. Pass1d populates this dotdot linklist. But it never frees this space. During the whole fsck runif it requires some change in structural files, it will do rebuild. Every time it rebuilds, it will add up the space to the already consumed memory and this way the total memory consumption will be huge.RESOLUTION:Code changes are done to free the dotdot list.* 4162687 (Tracking ID: 4145203)SYMPTOM:vxfs startup scripts fails to invoke veki for kernel version higher than 3.DESCRIPTION:vxfs startup script failed to start Veki, as it was calling system V init script to start veki instead of the systemctl interface.RESOLUTION:Current code changes checks if kernel version is greater than 3.x and if systemd is present then use systemctl interface otherwise use system V interface* 4162690 (Tracking ID: 4134661)SYMPTOM:Hang seen in the cp command in case of checkpoint promote in cluster filesystem environment.DESCRIPTION:The Hang is seen in cp command as we are not able to pull the inode blocks which is marked as overlay, hence made the code changes to pull the inode blocks marked as overlay.RESOLUTION:The Hang is seen in cp command as we are not able to pull the inode blocks which is marked as overlay, hence made the code changes to pull the inode blocks marked as overlay.* 4162691 (Tracking ID: 4084239)SYMPTOM:In case of OOM (Out of Memory) situation we might hit the issue if IOCTL fails to copy the data.DESCRIPTION:In case of error while copying the data (here it is OOM), we tried to release the lock which was never taken because of error.RESOLUTION:Fixed the bug with code changes.* 4162692 (Tracking ID: 4092440)SYMPTOM:# /opt/VRTS/bin/fsppadm enforce /mnt4UX:vxfs fsppadm: ERROR: V-3-27988: Placement policy file does not exist for mount point /mnt4: No such file or directory# echo $?0DESCRIPTION:FSPPADM command was returning rc 0 even in case of error during policy enformentmet.RESOLUTION:Fixed the issue by code change.* 4162693 (Tracking ID: 4119961)SYMPTOM:Machine hit with Kernel PANIC, and generated the core dump.DESCRIPTION:During read we were trying to release the lock which was never taken.RESOLUTION:Fixed the issue with code changes.* 4162694 (Tracking ID: 4099740)SYMPTOM:While mounting a file system, it fails with EBUSY error. Although on the setup, same device can not be seen as "mounted".DESCRIPTION:During mounting a filesystem, if it encounters error in kernel space, it leaks a hold count of the block device. This falsely implies the block device is still open in any future mounts. Because of that, when user retries the mount, the mount fails with EBUSY. It also causes memory leak for the same reason.RESOLUTION:Code changes are done to release the hold count on the block device properly.* 4162695 (Tracking ID: 4070819)SYMPTOM:Handling the case where the error is returned while we are going to get the inode from the last clone which is marked as overlay and is going to be removed.DESCRIPTION:We are going to get the inode which is marked as overlay from the last clone that is marked for deletion. Hence have made the code changes to handle the scenario where we can get the error while fetching the inode in this case.RESOLUTION:Have handled this scenario through code.* 4162696 (Tracking ID: 4058153)SYMPTOM:# mkfs.vxfs -o inosize=512 /dev/vx/dsk/testdg/testvol# mount.vxfs /dev/vx/dsk/testdg/testvol /mnt1# mkdir /mnt1/dir1nxattrset -n ab -v 012345678901234567890123456789012345678901234567890123456789012345678901 /mnt1/dir1 >>>>>> creating 88 byte nxattr# ./create_20k_file.sh >>>>>>>>>>>> creating 20k files inside /mnt1/dir1/ to create LDH attribute.Now if we remove LDH attain with some free inode, the fsck will go to invite loop.DESCRIPTION:We were doing calculation error while clearing the LDH attribute from inode.RESOLUTION:Fixed the bug with code changes, now FSCK will not hang and will clear the LDH attribute.* 4162697 (Tracking ID: 4136235)SYMPTOM:System with higher number of attribute inodes and pnlct inodes my see higher number of IOs on an idle system.DESCRIPTION:System with higher number of attribute inodes and pnlct inodes my see higher number of IOs on an idle CFS. Hence reducing the pnlct merge frequency may showsome performance improvement.RESOLUTION:Module parameter to change pnlct merge frequency.* 4162698 (Tracking ID: 4134194)SYMPTOM:vxfs/glm worker thread panic with kernel NULL pointer dereferenceDESCRIPTION:In vx_dir_realloc(), When the directory block is full, to fit new file entry it reallocate this directory block into a larger extent.So as the new extent gets allocated, the old cbuf is now part of the new extent.But we dont invalidate old cbuf during dir_realloc, which ends up with a staled cbuf in the cache.This staled buffer can cause the buffer overflow issue.RESOLUTION:Code changes are done to invalidate the cbuf immediately after the realloc.* 4162699 (Tracking ID: 4112056)SYMPTOM:Will have incorrect values in inode fields i_acl and i_default_acl that is 0, however expected value is ACL_NOT_CACHED (-1)DESCRIPTION:VxFS does not set get_acl() callback in inode_operations (i_op), hence whenever kernel (version 4.x and above) checks the presence of this callback and does not find, it sets i_acl and i_default_act fields to 0.RESOLUTION:Corrected the bug with code changes.* 4162700 (Tracking ID: 4068548)SYMPTOM:Fullfsck set on the file system and message "WARNING: msgcnt 222 mesg 017: V-2-17: vx_nattr_dirremove_1 - <mntpt> file system inode <ino> marked bad incore" logged in dmesgDESCRIPTION:In case if file system is full, allocation fails with ENOSPC. enospc processing is done through inactive_process. If the file creation is done by non-root user and this thread itself starts doing the worklist processing it may fail with EACCES while processing IEREMOVE on files with root ownership.RESOLUTION:Set the root credentials while doing enospc processing and restore the old credentials after it is done.* 4162701 (Tracking ID: 4137040)SYMPTOM:System got hung due to missing unlock on a file directory, this issue could be hit if there are a lot of mv(1) operations happening against one large VxFS directory.DESCRIPTION:In a large VxFS directory, LDH (alternate indexing) is activated once the number of directory entries cross the large directory threshold (vx_dexh_sz), LDH creates hash attribute inode into the main directory. The exclusive lock of this LDH inode is required during file system rename operations (mv(1)), in case of multiple rename operations happening against one large directory, the trylock of LDH inode may fail due to the contention, and VX_EDIRLOCK is returned.In case of VX_EDIRLOCK, VxFS should release the exclusive lock of source directory and update the locker list, then retry the operation. However VxFS releases the exclusive lock of target directory wrongly, instead of source and doesnt update the locker list, during the retry operation, although it happens to release the lock (target equals source if rename happens within the same dir), the locker list isnt updated, this locker record still remains in locker list, consequently, the same lock will not get released due to this extra record.RESOLUTION:Release the source dir lock instead of target, and update locker list accordingly.* 4162702 (Tracking ID: 4101075)SYMPTOM:Will hit the core dump with debug bits, during finding big size extent.DESCRIPTION:During search of big extent (32K) with delegation, it is okay for smap to be NULL, if it is not NULL then unlock it.RESOLUTION:Relaxed the assert as it was unnecessarily complaining, also added the code to free the smap if it is not NULL.* 4162703 (Tracking ID: 4132435)SYMPTOM:Failures seen in FSQA cmds->fsck tests, panic in get_dotdotlstDESCRIPTION:The inode getting processed in pass_unload->clean_dotdotlst(),was not in the incore imap table, so its related dotdot list is also not created.Because the dotdotlist is not initialized it hit the null pointerdereference error in clean_dotdotlst, hence the panic.RESOLUTION:Code changes are done to check for inode allocation status in incore imap table while cleaning the dotdot list.* 4162704 (Tracking ID: 4068953)SYMPTOM:# /opt/VRTS/bin/fsck /dev/vx/rdsk/testdg/testvol# mount.vxfs /dev/vx/dsk/testdg/testvol /testfsck# vxupgrade -n 17 /testfsck# umount /testfsck # /opt/VRTS/bin/fsck -o full -n /dev/vx/rdsk/testdg/testvolpass0 - checking structural filespass1 - checking inode sanity and blockspass2 - checking directory linkagepass3 - checking reference countspass4 - checking resource mapsfileset 1 au 0 imap incorrect - fix (ynq)n >>>>>>> NOT EXPECTEDfileset 1 iau 0 summary incorrect - fix? (ynq)n >>>>>>> NOT EXPECTEDOK to clear log? (ynq)nDESCRIPTION:In case of HOLE in ilist file we might hit the issue, because of incorrect calculation of available space.RESOLUTION:With the code changes, corrected the way we were calculating the space.* 4162706 (Tracking ID: 4117342)SYMPTOM:System might panic due to hard lock up detected on CPUDESCRIPTION:When purging the dentries, there is a possible race which can lead to corrupted vnode flag. Because of these corrupted flag, vxfs tries to purge dentry again and it gets stuck for vnode lock which was taken in the current thread context which leads to deadlock/softlockup.RESOLUTION:Code is modified to protect vnode flag with vnode lock.* 4162707 (Tracking ID: 4126943)SYMPTOM:Create lost+found directory in VxFS file system with default ACL permissions as 700.DESCRIPTION:Due to security reasons, there was ask to create lost+found directory in VxFS file system with default ACL permissions as 700. So that, except root, no other users are able to access files under lost+found directory.RESOLUTION:VxFS filesystem creation with mkfs command will now result in creation of lost+found directory with default ACL permissions as 700.* 4162708 (Tracking ID: 4134884)SYMPTOM:After unmounting the FS, when the diskgroup deport is initiated, it gives below error: vxvm:vxconfigd: V-5-1-16251 Disk group deport of testdg failed with error 70 - Volume or plex device is open or attachedDESCRIPTION:During mount of a dirty file system, vxvm device open count is leaked, and consequently, the deport of the vxvm DG got failed. During the VXFS FS mount operation the corresponding vxvm device will be opened.If the FS is not clean, it signifies mount to do the log replay. Later the log replay completes, and the mount will succeed.But this device open count leak causes the diskgroup deport to fail.RESOLUTION:Code changes are done to address the device open count leak.* 4162709 (Tracking ID: 4090088)SYMPTOM:while panic_on_warn is set, force umount might crash the server with following stacktrace: PID: 627188 TASK: ffff901777e89c00 CPU: 7 COMMAND: "vxumount" #0 machine_kexec at ffffffffb54653c8 #1 __crash_kexec at ffffffffb55af0cd #2 panic at ffffffffb5e3359f #3 __warn.cold at ffffffffb5e337c4 #4 report_bug at ffffffffb594501a #5 handle_bug at ffffffffb5e7be9c #6 exc_invalid_op at ffffffffb5e7c024 #7 asm_exc_invalid_op at ffffffffb6000a62 [exception RIP: blkdev_flush_mapping+252] #8 blkdev_put at ffffffffb58b41b0 #9 vx_force_umount at ffffffffc1675e8f [vxfs]#10 vx_aioctl_common at ffffffffc11dd80e [vxfs]#11 vx_aioctl at ffffffffc11d57db [vxfs]#12 vx_admin_ioctl at ffffffffc1550b2e [vxfs]#13 vxportalunlockedkioctl at ffffffffc0a61399 [vxportal]#14 __x64_sys_ioctl at ffffffffb578aca2#15 do_syscall_64 at ffffffffb5e7bb2bDESCRIPTION:Mode is not properly setting with FMODE_EXCL flag.RESOLUTION:Code changes have been checked-in to properly set the flag value.* 4162710 (Tracking ID: 4068201)SYMPTOM:File system corruptionDESCRIPTION:In certain cases we are not not flushing the intent log for transactions committed by a msg handler thread before replying to the msg.RESOLUTION:Flush the transactions done during ilist pull and push in case of error before sending the response.* 4162711 (Tracking ID: 4149899)SYMPTOM:Veritas file replication failover fails to perform the failover of job to target cluster.DESCRIPTION:As part of Veritas file failover operation, target filesystem needs to offline and made online to become new source.. After this operation the filesystem needs to marked with protected flag off, as this becomes new site and application can write to this site. So to update the flag current code performs second remount of file system which is taking time as it is processing file set removal as part of first offline(umount) and online(mount).RESOLUTION:Code changes have been done to make sure all operations including setting protected flag during single offline and online processing of filesystem* 4162835 (Tracking ID: 4142555)SYMPTOM:Modify reference to running fsck -y in mount.vxfs error message and update fsck_vxfs manpageDESCRIPTION:While trying to mount the corrupted FS the error message promts in to run fsck -y. Without understanding the implication of running fsck -y on a FS can lead to data loss.RESOLUTION:Updated the mount.vxfs error messages with recommedation to refer to fsck_vxfs manpage.And in the fsck_vxfs manpage added additional message to connect with veritas support for further assistance in collecting more debug logs before running fsck -y.* 4162853 (Tracking ID: 4158757)SYMPTOM:VxFS module failed to load on RHEL-8.10 kernel.DESCRIPTION:This issue occurs due to changes in the RHEL-8.10 kernel.RESOLUTION:VxFS module is updated to accommodate the changes in the kernel and load as expected on RHEL-8.10 kernel.* 4163378 (Tracking ID: 4075251)SYMPTOM:Performance bottleneck is seen in internal sequential write when there is very less space available in the filesystem.DESCRIPTION:If the filesystem doesn't have enough space left, then dalloc is turned off from vx_write_advise(). The code path: vx_write_advise() -> vx_dalloc_reserve() -> vx_space_reserve() -> vx_dalloc_checkdev_res()After turning off the dalloc, the writer goes through the slow path and tries to allocate the required space. As the FS doesn't have enough space, it gets ENOSPC and it initiates vx_dalloc_freeze() to process all pending allocations of dalloc inodes. This operation is a costly operation as there can be a huge number of inodes in the dalloc list. As this operation includes allocation and flushing both, so it can take a significant amount of time which directly impacts the initiator of vx_dalloc_freeze() which is a writer in this case. As the writer holds rwlock of an inode, all other writers for that inode will be blocked till the completion of vx_dalloc_freeze(). This causes a drop in IOPS and also reduces the write bandwidth.RESOLUTION:Set "VX_DALLOC_DISENOSPC" flag on fsext if free space of the filesystem falls below "fse_dalloc_eno_pauselim" and clears the flag incase the free space reaches "fse_dalloc_eno_reslim" to reenable dalloc.* 4163379 (Tracking ID: 4089199)SYMPTOM:Dynamic reconfiguration operation for CPU takes a lot of time. Temporary I/O hang is also observed during DR.DESCRIPTION:DR processing in VxFS is done for each CPU change notified by kernel. DR processing involves VxFS reinit and cluster-wide file system freeze.If the processor has SMT enabled then the cluster-wide file system freeze happens for each SMT thread per virtual CPU. This causes the slowness and temporary I/O hangs during CPU DR operations.RESOLUTION:Optimised the DR code to do the processing of several CPU DR events together.* 4163383 (Tracking ID: 4155961)SYMPTOM:System panic due to null i_fset in vx_rwlock().DESCRIPTION:Panic in vx_rwlock due to race between vx_rwlock() and vx_inode_deinit() function.Panic stack[exception RIP: vx_rwlock+174]..#10 __schedule#11 vx_write#12 vfs_write#13 sys_pwrite64#14 system_call_fastpathRESOLUTION:Code changes have been done to fix this issue.* 4163384 (Tracking ID: 4119281)SYMPTOM:Higher page-in requests on Solaris 11 SPARC.DESCRIPTION:After upgrading Infoscale, page-in requests are much higher. "vmstat" output looks normal but "sar" output looks abnormal (showing high page-in requests). "sar" is taking absolute sample for some reasons. "sar" is not supposed to use these values.RESOLUTION:Code changes are done to solve this issue* 4164534 (Tracking ID: 4129680)SYMPTOM:VxFS rpm does not have changelogDESCRIPTION:Changelog in rpm will help to find missing incidents with respect to other version.RESOLUTION:Changelog is generated and added to VxFS rpm.* 4164961 (Tracking ID: 4163337)SYMPTOM:Intermittent df slowness seen across cluster due to slow cluster-wide file system freeze.DESCRIPTION:For certain workload, intent log reset can happen relatively frequently and whenever it happens it will trigger cluster-wide freeze. If there are a lot of dirty buffers that need flushing and invalidation, then the freeze might take long time to finish. The slowest part in the invalidation of cluster buffers is the de-initialisation of its glm lock which requires lots of lock release messages to be sent to the master lock node. This can cause flowcontrol to be set at LLT layer and slow down the cluster-wide freeze and block commands like df, ls for that entire duration.RESOLUTION:Code is modified to avoid buffer flushing and invalidation in case freeze is triggered by intent log reset.* 4168371 (Tracking ID: 4162316)SYMPTOM:System PANICDESCRIPTION:CrowdStrike falcon might generate Kernel PANIC, during migration from native FS to VxFS.RESOLUTION:Required fix is added to vxfs code.* 4174512 (Tracking ID: 4168443)SYMPTOM:System panicked at vx_clonemap after a smap corruption. Following is the panic stack[exception RIP: vx_clonemap+317]#0 vx_tflush_map at ffffffffc0dd7b58 [vxfs]#1 vx_fsq_flush at ffffffffc0ff05c9 [vxfs]#2 vx_fsflush_fsq at ffffffffc0ff2c5f [vxfs]#3 vx_workitem_process at ffffffffc0eef9ea [vxfs]#4 vx_worklist_process at ffffffffc0ef8775 [vxfs]#5 vx_worklist_thread at ffffffffc0ef8828 [vxfs]#6 vx_kthread_init at ffffffffc0f7f8e4 [vxfs]#7 kthread at ffffffff810b3fb1#8 kthread_create_on_node at ffffffff810b3ee0#9 ret_from_fork at ffffffff816c0537DESCRIPTION:There was no proper error handling in case a smap is marked bad while changing its state in vx_smapchange. This can lead to a situation where system panic can occur while trying to flush an emap as the emap buffer will not be present incoreRESOLUTION:Code changes have been done to handle smap mapbad error properly.* 4174514 (Tracking ID: 4171368)SYMPTOM:Node panicked while unmounting filesystem.DESCRIPTION:panic in iput() due to invalid address in i_sb.If a nested unmount is stuck and the parent mount is unmounted simultaneously, the parent will be force unmounted, and its superblock will be cleared. This superblock address may then be reused and freed by another module, causing the i_sb to have an invalid address.iput vx_os_iput_enqueue [vxfs]vx_do_unmount [vxfs]vx_unmount [vxfs]generic_shutdown_super kill_block_super vx_kill_sb [vxfs]RESOLUTION:Code changes have been made to resolve this issue.Patch ID: VRTSvxfs-8.0.0.3200* 4096656 (Tracking ID: 4090032)SYMPTOM:System might panic in vx_dev_strategy() while Sybase or Oracle configuration, the panic stack looks like following:vx_dev_strategyvx_dio_physiovx_dio_rdwrivx_write_directvx_write1vx_write_common_slowvx_write_commonvx_writefop_writepwriteDESCRIPTION:When we are allocating a different buffer, vx_dev_strategy() unable to find the LDI handle.RESOLUTION:Code is modified to fix this issue.* 4119279 (Tracking ID: 4119281)SYMPTOM:Higher page-in requests on Solaris 11 SPARC.DESCRIPTION:After upgrading Infoscale, page-in requests are much higher. "vmstat" output looks normal but "sar" output looks abnormal (showing high page-in requests). "sar" is taking absolute sample for some reasons. "sar" is not supposed to use these values.RESOLUTION:Code changes are done to solve this issue* 4126124 (Tracking ID: 4121691)SYMPTOM:Unknown messages regarding udev rules for ignore_device are observed in /var/log/messages .systemd-udevd[841]: Configuration file /etc/udev/rules.d/60-vxca.rules is marked executable. Please remove executable permission bits. Proceeding anyway.DESCRIPTION:Unknown messages regarding udev rules for ignore_device are observed in /var/log/messages .systemd-udevd[841]: Configuration file /etc/udev/rules.d/60-vxca.rules is marked executable. Please remove executable permission bits. Proceeding anyway.RESOLUTION:Required changes have been done to handle this defect.* 4136241 (Tracking ID: 4136110)SYMPTOM:cmd "umount -l" is unmounting mount points even after adding mntlock in sles12 and sles15.DESCRIPTION:cmd "umount -l" is unmounting mount points even after adding mntlock in sles12 and sles15 due to missing umount.vxfs binary in /sbin directory.RESOLUTION:Code changes have been done to add umount.vxfs binary in /sbin directory in sles12 and sles15.* 4144027 (Tracking ID: 4126957)SYMPTOM:If "fsadm -o mntunlock=<string> <mountpoint>" and "umount -f <mountpoint>" operations are run in parallel,system may crash with following stack: vx_aioctl_unsetmntlock+0xd3/0x2a0 [vxfs] vx_aioctl_vfs+0x256/0x2d0 [vxfs] vx_admin_ioctl+0x156/0x2f0 [vxfs] vxportalunlockedkioctl+0x529/0x660 [vxportal] do_vfs_ioctl+0xa4/0x690 ksys_ioctl+0x64/0xa0 __x64_sys_ioctl+0x16/0x20 do_syscall_64+0x5b/0x1b0DESCRIPTION:There is a race condition between these two operations, due to which by the time fsadm thread tries to accessFS data structure, it is possible that umount operation has already freed the structures, which leads to panic.RESOLUTION:As a fix, the fsadm thread first checks if the umount operation is in progress. If so, it fails rather than continuing.* 4144029 (Tracking ID: 4137040)SYMPTOM:System got hung due to missing unlock on a file directory, this issue could be hit if there are a lot of mv(1) operations happening against one large VxFS directory.DESCRIPTION:In a large VxFS directory, LDH (alternate indexing) is activated once the number of directory entries cross the large directory threshold (vx_dexh_sz), LDH creates hash attribute inode into the main directory. The exclusive lock of this LDH inode is required during file system rename operations (mv(1)), in case of multiple rename operations happening against one large directory, the trylock of LDH inode may fail due to the contention, and VX_EDIRLOCK is returned.In case of VX_EDIRLOCK, VxFS should release the exclusive lock of source directory and update the locker list, then retry the operation. However VxFS releases the exclusive lock of target directory wrongly, instead of source and doesnt update the locker list, during the retry operation, although it happens to release the lock (target equals source if rename happens within the same dir), the locker list isnt updated, this locker record still remains in locker list, consequently, the same lock will not get released due to this extra record.RESOLUTION:Release the source dir lock instead of target, and update locker list accordingly.* 4144042 (Tracking ID: 4112056)SYMPTOM:Will have incorrect values in inode fields i_acl and i_default_acl that is 0, however expected value is ACL_NOT_CACHED (-1)DESCRIPTION:VxFS does not set get_acl() callback in inode_operations (i_op), hence whenever kernel (version 4.x and above) checks the presence of this callback and does not find, it sets i_acl and i_default_act fields to 0.RESOLUTION:Corrected the bug with code changes.* 4144043 (Tracking ID: 4126943)SYMPTOM:Create lost+found directory in VxFS file system with default ACL permissions as 700.DESCRIPTION:Due to security reasons, there was ask to create lost+found directory in VxFS file system with default ACL permissions as 700. So that, except root, no other users are able to access files under lost+found directory.RESOLUTION:VxFS filesystem creation with mkfs command will now result in creation of lost+found directory with default ACL permissions as 700.* 4144059 (Tracking ID: 4134661)SYMPTOM:Hang seen in the cp command in case of checkpoint promote in cluster filesystem environment.DESCRIPTION:The Hang is seen in cp command as we are not able to pull the inode blocks which is marked as overlay, hence made the code changes to pull the inode blocks marked as overlay.RESOLUTION:The Hang is seen in cp command as we are not able to pull the inode blocks which is marked as overlay, hence made the code changes to pull the inode blocks marked as overlay.* 4144060 (Tracking ID: 4132435)SYMPTOM:Failures seen in FSQA cmds->fsck tests, panic in get_dotdotlstDESCRIPTION:The inode getting processed in pass_unload->clean_dotdotlst(),was not in the incore imap table, so its related dotdot list is also not created.Because the dotdotlist is not initialized it hit the null pointerdereference error in clean_dotdotlst, hence the panic.RESOLUTION:Code changes are done to check for inode allocation status in incore imap table while cleaning the dotdot list.* 4144061 (Tracking ID: 4092440)SYMPTOM:# /opt/VRTS/bin/fsppadm enforce /mnt4UX:vxfs fsppadm: ERROR: V-3-27988: Placement policy file does not exist for mount point /mnt4: No such file or directory# echo $?0DESCRIPTION:FSPPADM command was returning rc 0 even in case of error during policy enformentmet.RESOLUTION:Fixed the issue by code change.* 4144063 (Tracking ID: 4116887)SYMPTOM:Running fsck -y on large size metasave with lots of hardlinks is consuming huge amount of system memory.DESCRIPTION:On a FS with lot of hardlinks, requires a lot of memory for storing dotdot information in memory. Pass1d populates this dotdot linklist. But it never frees this space. During the whole fsck runif it requires some change in structural files, it will do rebuild. Every time it rebuilds, it will add up the space to the already consumed memory and this way the total memory consumption will be huge.RESOLUTION:Code changes are done to free the dotdot list.* 4144074 (Tracking ID: 4117342)SYMPTOM:System might panic due to hard lock up detected on CPUDESCRIPTION:When purging the dentries, there is a possible race which can lead to corrupted vnode flag. Because of these corrupted flag, vxfs tries to purge dentry again and it gets stuck for vnode lock which was taken in the current thread context which leads to deadlock/softlockup.RESOLUTION:Code is modified to protect vnode flag with vnode lock.* 4144082 (Tracking ID: 4134194)SYMPTOM:vxfs/glm worker thread panic with kernel NULL pointer dereferenceDESCRIPTION:In vx_dir_realloc(), When the directory block is full, to fit new file entry it reallocate this directory block into a larger extent.So as the new extent gets allocated, the old cbuf is now part of the new extent.But we dont invalidate old cbuf during dir_realloc, which ends up with a staled cbuf in the cache.This staled buffer can cause the buffer overflow issue.RESOLUTION:Code changes are done to invalidate the cbuf immediately after the realloc.* 4144086 (Tracking ID: 4136235)SYMPTOM:System with higher number of attribute inodes and pnlct inodes my see higher number of IOs on an idle system.DESCRIPTION:System with higher number of attribute inodes and pnlct inodes my see higher number of IOs on an idle CFS. Hence reducing the pnlct merge frequency may showsome performance improvement.RESOLUTION:Module parameter to change pnlct merge frequency.* 4144117 (Tracking ID: 4099740)SYMPTOM:While mounting a file system, it fails with EBUSY error. Although on the setup, same device can not be seen as "mounted".DESCRIPTION:During mounting a filesystem, if it encounters error in kernel space, it leaks a hold count of the block device. This falsely implies the block device is still open in any future mounts. Because of that, when user retries the mount, the mount fails with EBUSY. It also causes memory leak for the same reason.RESOLUTION:Code changes are done to release the hold count on the block device properly.* 4144119 (Tracking ID: 4134884)SYMPTOM:After unmounting the FS, when the diskgroup deport is initiated, it gives below error: vxvm:vxconfigd: V-5-1-16251 Disk group deport of testdg failed with error 70 - Volume or plex device is open or attachedDESCRIPTION:During mount of a dirty file system, vxvm device open count is leaked, and consequently, the deport of the vxvm DG got failed. During the VXFS FS mount operation the corresponding vxvm device will be opened.If the FS is not clean, it signifies mount to do the log replay. Later the log replay completes, and the mount will succeed.But this device open count leak causes the diskgroup deport to fail.RESOLUTION:Code changes are done to address the device open count leak.* 4145797 (Tracking ID: 4145203)SYMPTOM:vxfs startup scripts fails to invoke veki for kernel version higher than 3.DESCRIPTION:vxfs startup script failed to start Veki, as it was calling system V init script to start veki instead of the systemctl interface.RESOLUTION:Current code changes checks if kernel version is greater than 3.x and if systemd is present then use systemctl interface otherwise use system V interface* 4149436 (Tracking ID: 4141665)SYMPTOM:Security vulnerabilities exist in the Zlib third-party components used by VxFS.DESCRIPTION:VxFS uses Zlib third-party components with some security vulnerabilities.RESOLUTION:VxFS is updated to use a newer version of Zlib third-party components in which the security vulnerabilities have been addressed.Patch ID: VRTSvxfs-8.0.0.3100* 4154855 (Tracking ID: 4141665)SYMPTOM:Security vulnerabilities exist in the Zlib third-party components used by VxFS.DESCRIPTION:VxFS uses Zlib third-party components with some security vulnerabilities.RESOLUTION:VxFS is updated to use a newer version of Zlib third-party components in which the security vulnerabilities have been addressed.Patch ID: VRTSvxfs-8.0.0.2900* 4092518 (Tracking ID: 4096267)SYMPTOM:Veritas File Replication jobs might failed when there are large number of jobs run in parallel.DESCRIPTION:File Replication Jobs might fail, with Large number of jobs configured and running in parallel with Veritas File Replication. With large number of jobs there is a chance of referring a job which is already freed, due to which there is a core generated with replication service and job might failed.RESOLUTION:updated code to handle the code to take a hold while checking invalid job configuration.* 4097466 (Tracking ID: 4114176)SYMPTOM:After failover, job sync fails with error "Device or resource busy".DESCRIPTION:If job is in failed state on target because of job failure from source side, repld was not updating its state when it was restarted in recovery mode. Because of which job state was remaining in running state even after successful replication on target. With this state on target, if job is promoted, then replication process was not creating new ckpt for first sync after failover which was resulting in corrupting state file on new source. Because of this incorrect/corrupt state file, job sync from new source was failing with error "Device or resource busy".RESOLUTION:Code is modified to correct the state on target when job was started in recovery mode.* 4107367 (Tracking ID: 4108955)SYMPTOM:VFR job hangs on source if thread creation fails on target.DESCRIPTION:On Target, if thread creation for pass completion fails because of high memory usage, repld demon doesn't send that failure reply to source. This can lead to vxfsreplicate process to remains in waiting state indefinitely for reply for pass completion from target. This will lead to job hang on source and will need manual intervention to kill the job.RESOLUTION:Code is modified to retry thread creation on target and if it fails after 5 retries, target will reply to source with appropriate error.* 4111457 (Tracking ID: 4117827)SYMPTOM:Without tunable change the logfile permission will always be 600 EO compliantDESCRIPTION:Tunable values and behavior: Value Behavior 0 (default) 600 permissions, update existing file permissions on upgrade 1 640 permissions, update existing file permissions on upgrade 2 644 permissions, update existing file permissions on upgrade 3 Inherit umask, update existing file permissions on upgrade 10 600 permissions, dont touch existing file permissions on upgrade 11 640 permissions, dont touch existing file permissions on upgrade 12 644 permissions, dont touch existing file permissions on upgrade 13 Inherit umask, dont touch existing file permissions on upgrade -------------------------------------------------------------------------------------- Adding new tunable as part of vxtunefs command which is per-node global tunable (not per filesystem). For Executive Order, CPI will be having workflow to update the tunable during installation/upgrade/configuration which will take care of updating in all nodes.RESOLUTION:New tunable is added to vxtunefs command. How to set tunable: /opt/VRTS/bin/vxtunefs -D eo_perm=1* 4112417 (Tracking ID: 4094326)SYMPTOM:mdb invocation displays message "failed to add vx_sl_node_level walker: walk name already in use"DESCRIPTION:In vx_sl_kmcache_init(), kmcache is initialized for each level (in this case it is 8) separately. For passing the cache name as an argument to kmem_cache_create(), we have used a macro. #define VX_SL_KMCACHE_NAME(level) "vx_sl_node_"#level #define VX_SL_KMCACHE_CREATE(level) \ kmem_cache_create(VX_SL_KMCACHE_NAME(level), \ VX_KMEM_SIZE(VX_SL_KMCACHE_SIZE(level)),\ 0, NULL, NULL, NULL, NULL, NULL, 0); While using this macro, we have passed "level" as an argument and that has been expanded as "vx_sl_node_level" for all the 8 levels in `for` loop. This is causing the cache allocation for all the 8 levels with same name.RESOLUTION:Passing separate variable value (as level value) to VX_SL_KMCACHE_NAME as it is done in vx_wb_sl_kmcache_init().* 4118795 (Tracking ID: 4100021)SYMPTOM:Running setfacl followed by getfacl resulting in "No such device or address" error.DESCRIPTION:When running setfacl command on some of the directories which have the VX_ATTR_INDIRECT type of acl attribute, it is not removing the existing acl attribute and adding a new one, which should not happen ideally. This is resulting in the failure of getfacl with following "No such device or address" error.RESOLUTION:we have done the code chages to removal of VX_ATTR_INDIRECT type acl in setfacl code.* 4119023 (Tracking ID: 4116329)SYMPTOM:fsck -o full -n command will fail with error: "ERROR: V-3-28446: bc_write failure devid = 0, bno = 8, len = 1024"DESCRIPTION:Previously, to correct the file system WORM/SoftWORM, we didn't check if user wanted to correct the pflags or just wanted to validate if value is flag is missing or not. Also fsck was not capable to handle SOFTWORM flag.RESOLUTION:Code added to not try to fix the the problem if user ran fsck with -n option. Also SOFTWORM scenario is added.* 4123143 (Tracking ID: 4123144)SYMPTOM:fsck binary generating coredumpDESCRIPTION:In internal testing we found that fsck binary generates coredump due to below mentioned assert when we try to repair corrupted file system using below command: ./fsck -o full -y /dev/vx/rdsk/testdg/vol1 ASSERT(fset >= VX_FSET_STRUCT_INDEX)RESOLUTION:Added code to set default (primary) fileset by scanning the fset header list.Patch ID: VRTSvxfs-8.0.0.2700* 4113911 (Tracking ID: 4113121)SYMPTOM:The VxFS module fails to load on RHEL8.8.DESCRIPTION:This issue occurs due to changes in the RHEL8.8.RESOLUTION:Updated VXFS to support RHEL 8.8.* 4114019 (Tracking ID: 4067505)SYMPTOM:Fsck reports error invalid VX_AF_OVERLAY aflagsDESCRIPTION:If the inode does not have push linkage (inode not allocated / inode and data already pushed), we skip pushing the data blocks when the inode is removed. Inode will have overlay data blocks, gen bumped up and IEREMOVE set. During extop processing size is set to 0 and bmap is cleared. This is a valid scenario. Fsck while validating the inodes with overlay flag set, expects gen can be different only if the overlay inode has IEREMOVE set and it is last clone in the chain.RESOLUTION:If the push inode is not present allow gen to be different even if the clone is not last in chain.* 4114020 (Tracking ID: 4083056)SYMPTOM:Hang observed while punching the smaller hole over the bigger hole.DESCRIPTION:We observed the hang while punching the smaller hole over the bigger hole in the file due to the tight race while processing the punching of the hole to the file and flushing it to the disk.RESOLUTION:Code changes checked in.* 4114021 (Tracking ID: 4101634)SYMPTOM:Fsck reports error directory block containing inode has incorrect file-type and directory contains invalid directory blocks.DESCRIPTION:While doing diretory sanity in fsck we skip updating new directory type ondisk in case of filetype error, hence fsck reporting incorrect file-type error and directory contains invalid directory blocks .RESOLUTION:While doing diretory sanity in fsck updating new directory type ondisk in case of filetype error.Patch ID: VRTSvxfs-8.0.0.2600* 4114654 (Tracking ID: 4114652)SYMPTOM:The VxFS module fails to load on RHEL8.7 minor kernel 4.18.0-425.19.2.DESCRIPTION:This issue occurs due to changes in the RHEL8.7 minor kernel.RESOLUTION:Updated VXFS to support RHEL 8.7 minor kernel 4.18.0-425.19.2.Patch ID: VRTSvxfs-8.0.0.2300* 4108381 (Tracking ID: 4107777)SYMPTOM:The VxFS module fails to load on RHEL8.7 minor kernel.DESCRIPTION:This issue occurs due to changes in the RHEL8.7 minor kernel.RESOLUTION:Modified existing modinst-vxfs script to accommodate the changes in the kernel and load the correct module.Patch ID: VRTSvxfs-8.0.0.2200* 4100925 (Tracking ID: 4100926)SYMPTOM:VxFS module failed to load on RHEL8.7DESCRIPTION:The RHEL8.7 is new release and it has some changes in kernel which caused VxFS module failed to load on it.RESOLUTION:Added code to support VxFS on RHEL8.7.Patch ID: VRTSvxfs-8.0.0.2100* 4095889 (Tracking ID: 4095888)SYMPTOM:Security vulnerabilities exist in the Sqlite third-party components used by VxFS.DESCRIPTION:VxFS uses the Sqlite third-party components in which some security vulnerability exist.RESOLUTION:VxFS is updated to use newer version of this third-party components in which the security vulnerabilities have been addressed.Patch ID: VRTSvxfs-8.0.0.1800* 4068960 (Tracking ID: 4073203)SYMPTOM:Veritas file replication might generate a core while replicating the files to target when rename and unlink operation is performed on a file with FCL( file change log) mode on.DESCRIPTION:vxfsreplicate process of Veritas file replicator might get a segmentation fault with File change mode on when rename and unlink operation are performed on a file.RESOLUTION:Addressed the issue to replicate the files, in scenarios involving rename and unlink operation with FCL mode on.* 4071108 (Tracking ID: 3988752)SYMPTOM:Use ldi_strategy() routine instead of bdev_strategy() for IO's in solaris.DESCRIPTION:bdev_strategy() is deprecated from solaris code and was causing performance issues when used for IO's. Solaris has recommended to use LDI framework for all IO's.RESOLUTION:Code is modified to use ldi framework for all IO's in solaris.* 4072228 (Tracking ID: 4037035)SYMPTOM:VxFS should have the ability to control the number of inactive processing threads.DESCRIPTION:VxFS may spawn a large number of worker threads that become inactive over time. As a result, heavy lock contention occurs during the removal of inactive threads on high-end servers.RESOLUTION:To avoid the contention, a new tunable, vx_ninact_proc_threads, is added. You can use vx_ninact_proc_threads to adjust the number of inactive processing threads based on your server configuration and workload.* 4078335 (Tracking ID: 4076412)SYMPTOM:Addressing Executive Order (EO) 14028, initial requirements which is intended to improve the Federal Governments investigative and remediation capabilities related to cybersecurity incidents. Executive Order helps in improving the nation's cybersecurity and also enhance any organization's cybersecurity and software supply chain integrity.DESCRIPTION:Executive Order helps in improving the nation's cybersecurity and also enhance any organization's cybersecurity and software supply chain integrity, some of the initial requirements will enable the logging which is compliant to Executive Order. This comprises of command logging, logging unauthorised access in filesystem and logging WORM events on filesystem. Also include changes to display IP address for Veritas File replication at control plane based on tunable.RESOLUTION:The initial requirements of EO are addressed in this release. As per Executive order(EO) for some of the requirements it should be Tunable based. For example IP logging where ever applicable (for VFR it should be at control plane(not for every data transfer), and this is also tunable based. Also for logging some kernel logs, like worm events(plan is to log those to syslog) etc are tunable based. Introduced new tunable, eo_logging_enable. There is a protocol change because of the introduction of the tunable. Though the changes are planned for TOT first and then will go to Update patch on 80all maint for EO release, there is impact of this protocol change for update patch. We might need to update protocol change with middle protocol version between existing protocol version and new protocol version(introduced because of eo) For VFR, IP addresses of source and destination are needed to be logged as part of EO. IP addresses will be included in the log while logging Starting/Resuming a job in VFR. Log Location: /var/VRTSvxfs/replication/log/mount_point-job_name.log There are 2 ways to fetch the IP address of the source and target. One is to get the IP addresses stored in the link structure of a session. These IPs are obtained by resolving the source and target hostname. It may contain both IPv4 and IPv6 for a node, and we cannot speculate on which IP actual connection has happened. The second way is to get the socket descriptor from an active connection of the session. This socket descriptor can be used to fetch the source and target IP associated with it. The second method is seems to get the actual IP addresses used for the connection between source and target. The change contains to fetch IP addresses from socket descriptor after establishing connections. More details on EO Logging with respective handling for initial release for VxFS https://confluence.community.veritas.com/pages/viewpage.action?spaceKey=VEStitle=EO+VxFS+Scrum+Page* 4078520 (Tracking ID: 4058444)SYMPTOM:Loop mounts using files on VxFS fail on Linux systems running kernel version 4.1 or higher.DESCRIPTION:Starting with the 4.1 version of the Linux kernel, the driver loop.ko uses a new API for read and write requests to the file which was not previously implemented in VxFS. This causes the virtual disk reads during mount to fail while using the -o loop option , causing the mount to fail as well. The same functionality worked in older kernels (such as the version found in RHEL7).RESOLUTION:Implemented a new API for all regular files on VxFS, allowing usage of the loop device driver against files on VxFS as well as any other kernel drivers using the same functionality.* 4079142 (Tracking ID: 4077766)SYMPTOM:VxFS kernel module might leak memory during readahead of directory blocks.DESCRIPTION:VxFS kernel module might leak memory during readahead of directory blocks due to missing free operation of readahead-related structures.RESOLUTION:Code in readahead of directory blocks is modified to free up readahead-related structures.* 4079173 (Tracking ID: 4070217)SYMPTOM:Command fsck might fail with 'cluster reservation failed for volume' message for a disabled cluster-mounted filesystem.DESCRIPTION:On a disabled cluster-mounted filesystem, release of cluster reservation might fail during unmount operation resulting in a failure of command fsck with 'cluster reservation failed for volume' message.RESOLUTION:Code is modified to release cluster reservation in unmount operation properly even for cluster-mounted filesystem.* 4082260 (Tracking ID: 4070814)SYMPTOM:Security Vulnerability observed in Zlib a third party component VxFS uses.DESCRIPTION:In an internal security scans vulnerabilities in Zlib were found.RESOLUTION:Upgrading the third party component Zlib to address these vulnerabilities.* 4082865 (Tracking ID: 4079622)SYMPTOM:Migration uses normal read/write file operation instead of read/write iter functions. vxfs requires read/write iter functions from Linux kernel 5.14.DESCRIPTION:Starting with 5.14 version of the Linux kernel, vxfs uses a read/write iter file operation for migration.RESOLUTION:Developed a common function for read/write which get called for normal and iter read/write file operation.* 4083335 (Tracking ID: 4076098)SYMPTOM:FS migration from ext4 to vxfs on Linux machines with falcon-sensor enabled, may failDESCRIPTION:Falcon-sensor driver installed on test machines is tapping system calls such as close and is doing some additional vfs calls such as read. Due to this vxfs driver received read file - operation call from fsmigbgcp process context. Read operation is allowed only on special files from fsmigbgcp process context. Since the file in picture was not a special file, the vxfs debug code asserted.RESOLUTION:As a fix, we are now allowing the read on non special files from fsmigbgcp process context. [Note: - There were other related issues fixed in this incident. But those are not likely to be hit in customer environment as they are negative test scenarios (like trying to overwrite migration special file - deflist) and may not be relevant to customer. - I am not covering them in above* 4085623 (Tracking ID: 4085624)SYMPTOM:While running fsck with -o and full -y on corrupted FS, fsck may dump core.DESCRIPTION:Fsck builds various in-core maps based on on-disk structural files, one such map is dotdotmap (which stores info about parent directory). For regular fset (like 999), the dotdotmap is initialized only for primary ilist (inode list for regular inodes). It is skipped for attribute ilist (inode list for attribute inodes). This is because attribute inodes do not have parent directories as is the case for regular inodes. While attempting to resolve inconsistencies in FS metadata, fsck tries to clean up dotdotmap for attribute ilist. In the absence of a check, dotdotmap is re-initialized for attribute ilist causing segmentation fault.RESOLUTION:In the codepath where fsck attempts to reinitialize the dotdotmap, a check added to skip reinitialization of dotdotmap for attribute ilist.* 4085839 (Tracking ID: 4085838)SYMPTOM:Command fsck may generate core due to processing of zero size attribute inode.DESCRIPTION:Command fsck is modified to skip processing of zero size attribute inode.RESOLUTION:Command fsck fails due to allocation of memory and dereferencing it for zero size attribute inode.* 4086085 (Tracking ID: 4086084)SYMPTOM:VxFS mount operation causes system panic when -o context is used.DESCRIPTION:VxFS mount operation supports context option to override existing extended attributes, or to specify a different, default context for file systems that do not support extended attributes. System panic observed when -o context is used.RESOLUTION:Required code changes are added to avoid panic.* 4088341 (Tracking ID: 4065575)SYMPTOM:Write operation might be unresponsive on a local mounted VxFS filesystem in a no-space conditionDESCRIPTION:Write operation might be unresponsive on a local mounted VxFS filesystem in a no-space condition due to a race between two writer threads to take read-write lock the file to do a delayed allocation operation on it.RESOLUTION:Code is modified to allow thread which is already holding read-write lock to complete delayed allocation operation, other thread will skip over that file.Patch ID: VRTSvxfs-8.0.0.1700* 4081150 (Tracking ID: 4079869)SYMPTOM:Security Vulnerability found in VxFS while running security scans.DESCRIPTION:In our internal security scans we found some Vulnerabilities in VxFS third party components. The Attackers can exploit these security vulnerability to attack on system.RESOLUTION:Upgrading the third party components to resolve these vulnerabilities.* 4083948 (Tracking ID: 4070814)SYMPTOM:Security Vulnerability found in VxFS while running security scans.DESCRIPTION:In our internal security scans we found some Vulnerabilities in VxFS third party component Zlib.RESOLUTION:Upgrading the third party component Zlib to resolve these vulnerabilities.Patch ID: VRTSvxfs-8.0.0.1200* 4055808 (Tracking ID: 4062971)SYMPTOM:All the operations like ls, create are blocked on file systemDESCRIPTION:In WORM file system we do not allow directory rename. When partition directory is enabled, new directories are created and files are moved under this leaf directory based on hash. Due to WORM FS this rename operation was blocked and splitting could not complete. Blocking all the operations on file system.RESOLUTION:Allow directory renaming in the context of partition directory split and merge.* 4056684 (Tracking ID: 4056682)SYMPTOM:New features information on a filesystem with fsadm(file system administration utility) from a device is not displayed.DESCRIPTION:Information about new features like WORM (Write once read many), auditlog is correctly updated with a file system mounted through the fsadm utility, but on the underlying device the new feature information is not displayed.RESOLUTION:Updated fsadm utility to display the new feature information correctly.* 4062606 (Tracking ID: 4062605)SYMPTOM:Minimum retention time cannot be set if the maximum retention time is not set.DESCRIPTION:The tunable - minimum retention time cannot be set if the tunable - maximum retention time is not set. This was implemented to ensure that the minimum time is lower than the maximum time.RESOLUTION:Setting of minimum and maximum retention time is independent of each other. Minimum retention time can be set without the maximum retention time being set.* 4065565 (Tracking ID: 4065669)SYMPTOM:Creating non-WORM checkpoints fails when the tunables - minimum retention time and maximum retention time are set.DESCRIPTION:Creation of non-WORM checkpoints fails as all WORM-related validations are extended to non-WORM checkpoints also.RESOLUTION:WORM-related validations restricted to WORM fsets only, allowing non-WORM checkpoints to be created.* 4065651 (Tracking ID: 4065666)SYMPTOM:All the operations like ls, create are blocked on file system directory where there are WORM enabled files and retention period not expiredDESCRIPTION:For WORM file system, files whose retention period is not expired can not be renamed. When partition directory is enabled, new directories are created and files are moved under this leaf directory based on hash. Due to WORM FS this rename operation was blocked and splitting could not complete. Blocking all the operations on file system.RESOLUTION:Allow directory renaming of files even if retention period is not expired in the context of partition directory split and merge.Patch ID: VRTSvxfs-8.0.0.1100* 4061114 (Tracking ID: 4052883)SYMPTOM:The VxFS module fails to load on RHEL 8.5.DESCRIPTION:This issue occurs due to changes in the RHEL 8.5 kernel.RESOLUTION:VxFS module is updated to accommodate the changes in the kernel and load as expected on RHEL 8.5.Patch ID: VRTSdbed-8.0.0.1800* 4079372 (Tracking ID: 4073842)SYMPTOM:Oracle 21c is not supported on earlier product versions.DESCRIPTION:Implemented Oracle 21c support with Storage Foundation for Databases.RESOLUTION:Changes are done to support Oracle 21c with Storage Foundation for Databases.Patch ID: VRTSdbac-8.0.0.2200* 4108954 (Tracking ID: 4107779)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 7 GA kernel(4.18.0-425.3.1.el8.x86_64).RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7) is now introduced.Patch ID: VRTSdbac-8.0.0.2100* 4100204 (Tracking ID: 4100203)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 6.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7(RHEL8.7) is now introduced.Patch ID: VRTSdbac-8.0.0.1900* 4090090 (Tracking ID: 4090485)SYMPTOM:Installation of Oracle 12c GRID and database fails on RHEL8.*/OL8.* with GLIBC package errorDESCRIPTION:On RHEL8/OL8 with GLIBC version 2.2.5, VCSMM lib uses the available default version and hence fails to build with the following error message:INFO: /u03/app/12201/dbbase/dbhome/lib//libskgxn2.so: undefined reference to `memcpy@GLIBC_2.14' INFO: make: *** [/u03/app/12201/dbbase/dbhome/rdbms/lib/ins_rdbms.mk:1013: /u03/app/12201/dbbase/dbhome/rdbms/lib/orapwd] Error 1RESOLUTION:RHEL8/OL8 VCSMM module is built with GLIBC 2.2.5.Patch ID: VRTSdbac-8.0.0.1800* 4089728 (Tracking ID: 4089722)SYMPTOM:VRTSgab , VRTSamf and VRTSdbed driver does not load on RHEL and SLES platform.DESCRIPTION:Need recompilation of VRTSgab , VRTSamf and VRTSdbed with latest changes.RESOLUTION:Recompiled the VRTSgab , VRTSamf and VRTSdbed.Patch ID: VRTSdbac-8.0.0.1100* 4053178 (Tracking ID: 4053171)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 4.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 5(RHEL8.5) is now introduced.Patch ID: VRTSveki-8.0.0.2500* 4118568 (Tracking ID: 4110457)SYMPTOM:Veki packaging failure due to missing of storageapi specific filesDESCRIPTION:While creating the build area for different components like GLM, GMS, ORAODM, unixvm, VxFS veki build area creation were failing because of storageapi changes were not taken care in the Veki mk-symlink and build scripts.RESOLUTION:Added support for creation of storageapi build area, storageapi packaging changes via veki, and storageapi build via veki from Veki makefiles.This is helping to package the storageapi along with veki and resolving all interdependenciesPatch ID: VRTSveki-8.0.0.1800* 4056647 (Tracking ID: 4055072)SYMPTOM:Upgrading VRTSveki package using yum reports following error as "Starting veki /etc/vx/veki: line 51: [: too many arguments"DESCRIPTION:While upgrading VRTSveki package, presence of multiple module directories might result in upgrade script printing error message.RESOLUTION:Code is modified to check for specific module directory related to current kernel version in VRTSveki upgrade script.Patch ID: VRTSveki-8.0.0.1200* 4070027 (Tracking ID: 4066550)SYMPTOM:After reboot LLT, GAB service fail to start as Veki service start times out.DESCRIPTION:After reboot, when systemd tries to bring multiple services in parallel at the same time of Veki, Veki startup times out. The default Veki startup timeout was 90 seconds, which is getting increased to 300 seconds with this patchRESOLUTION:Increasing Veki start timeoutPatch ID: VRTSfsadv-8.0.0.2200* 4103001 (Tracking ID: 4103002)SYMPTOM:Replication failures observed in internal testingDESCRIPTION:Replication related code changes done in VxFS repository to fix replication failures. The replication binaries are part of VRTSfsadv.RESOLUTION:Compiled VRTSfsadv with VxFS changes.Patch ID: VRTSfsadv-8.0.0.2100* 4092150 (Tracking ID: 4088024)SYMPTOM:Security vulnerabilities exist in the OpenSSL third-party components used by VxFS.DESCRIPTION:VxFS uses the OpenSSL third-party components in which some security vulnerability exist.RESOLUTION:VxFS is updated to use newer version (1.1.1q) of this third-party components in which the security vulnerabilities have been addressed. To accommodate the changes vxfs_solutions is added with libboost_system entries in Makefile [dedup/pdde/sdk/common/Makefile].Patch ID: VRTSrest-2.0.0.1300* 4088973 (Tracking ID: 4089451)SYMPTOM:When a read-only filesystem was created on a Volume, GET on mountpoint's details was throwing error.DESCRIPTION:When a read-only filesystem was created on a Volume, GET on mountpoint's details was throwing error as the command which was being used was not working for read-only filesystem.RESOLUTION:Used the appropriate command to get details of mountpoint.* 4089033 (Tracking ID: 4089453)SYMPTOM:Some VCS REST APIs were crashing the Gunicorn worker.DESCRIPTION:Calling some VCS-related APIs were crashing the gunicorn worker handling the request. A new worker was automatically being spawned.RESOLUTION:Fixed the related foreign function call interface in the source code.* 4089041 (Tracking ID: 4089449)SYMPTOM:GET resources API on empty service group was throwing an error.DESCRIPTION:When GET resources API was called on an empty service group it was giving an error, and the scenario is not handled.RESOLUTION:Scenario added to the code to resolve the issue.* 4089046 (Tracking ID: 4089448)SYMPTOM:Logging in REST API was not in EO-compliant format.DESCRIPTION:Timestamp format is not EO-compliant and some attributes were missing for EO compliance.RESOLUTION:Changed timestamp, added new attributes like nodename, response time, source and destination IP addresses, username to REST server logs.Patch ID: -3.9.2.24* 4114375 (Tracking ID: 4113851)SYMPTOM:Open CVE's detected for the python programming language and other python modules being used in VRTSpythonDESCRIPTION:Some open CVE's are exploitable in VRTSpython for IS 8.0RESOLUTION:VRTSpython is patched with all the open CVE's which are impacting IS 8.0.Patch ID: -5.34.0.4* 4072234 (Tracking ID: 4069607)SYMPTOM:Security vulnerability detected on VRTSperl 5.34.0.0 released with Infoscale 8.0.DESCRIPTION:Security vulnerability detected in the Net::Netmask module.RESOLUTION:Upgraded Net::Netmask module and re-created VRTSperl 5.34.0.1 version to fix the vulnerability .* 4075150 (Tracking ID: 4075149)SYMPTOM:Security vulnerabilities detected in OpenSSL packaged VRTSperl/VRTSpython released with Infoscale 8.0.DESCRIPTION:Security vulnerabilities detected in the OpenSSL.RESOLUTION:Upgraded OpenSSL version and re-created VRTSperl/VRTSpython version to fix the vulnerability .Patch ID: VRTSsfmh-HF0800510* 4157270 (Tracking ID: 4157265)SYMPTOM:NADESCRIPTION:NARESOLUTION:NAPatch ID: -8.0.0.1700* 4117339 (Tracking ID: 4132683)SYMPTOM:Need a tool to collect IS product specific information from both standalone node and clustered env.DESCRIPTION:We don't have a tool that provides a consolidated report on all the InfoScale services/VCS information from different nodes of the cluster or from standalone server. we need to execute multiple commands separately to gather the information.RESOLUTION:A new tool has been added for collecting various product stats and service information that works across clustered and standalone environment. Password-less SSH should be configured among nodes from which script is running and nodes provided.* 4124177 (Tracking ID: 4125116)SYMPTOM:Logs collected from different tools of VRTSspt are stored at different location.DESCRIPTION:Currently FirstLook generates logs inside current working directory, CfsGather generates logs inside /opt/VRTSspt/FirstLook, vxgetcore generates logs inside /var/tmp and Collect_memstats_linux generates logs inside /tmp. So, consolidating the log location for VRTSspt tools to /var/log/VRTSspt.RESOLUTION:Logs collected from FirstLook, CfsGather, vxgetcore and Collect_memstats_linux will be stored at /var/log/VRTSspt location.* 4129889 (Tracking ID: 4114988)SYMPTOM:No Progress status reporting through long running metasave utilityDESCRIPTION:For large filesystems, collecting metadata through metasave utility might take a long time. Currently, it is a silent command that does not show the progress status of the collection being done during command execution.RESOLUTION:Code changes have been done to show the progress status while metadata collection is being done. Also, it initially estimates the total space requirement for metadata collection and if the required space is not available at the given location, metasave command would fail with ENOSPC.* 4139975 (Tracking ID: 4149462)SYMPTOM:New script is provided list_missing_incidents.py which compares changelogs of rpm and lists missing incidents in new version.DESCRIPTION:list_missing_incidents.py compares changelogs of old version rpm with new version rpm and lists missing incidents in new-version rpm if any. For details of script refer README.list_missing_incidents in VRTSspt packageRESOLUTION:list_missing_incidents.py compares changelogs of old version rpm with new version rpm and lists missing incidents in new-version rpm if any. For details of script refer README.list_missing_incidents in VRTSspt package* 4146957 (Tracking ID: 4149448)SYMPTOM:New script is provided check_incident_inchangelog.py which will check if incident abstract is present in changelog.DESCRIPTION:If Changelog is present in rpm or installed package, then provided script in VRTSspt can check if incident abstract is present in changelog. For details of script refer README.check_incident_inchangelog in VRTSspt packageRESOLUTION:If Changelog is present in rpm or installed package, then provided script in VRTSspt can check if incident abstract is present in changelog. For details of script refer README.check_incident_inchangelog in VRTSspt packagePatch ID: VRTSvcs-8.0.0.2700* 4161822 (Tracking ID: 4129493)SYMPTOM:Tenable security scan kills the Notifier resource.DESCRIPTION:When nmap port scan performed on port 14144 (on which notifier process is listening), notifier gets killed because of connection request.RESOLUTION:The required code changes have done to prevent Notifier agent crash when nmap port scan is performed on notifier port 14144.* 4162953 (Tracking ID: 4136359)SYMPTOM:When upgrading InfoScale with latest Public Patch Bundle or VRTSvcsag package, types.cf is updated.DESCRIPTION:To use new types, attribute(like PanicSystemOnVGLoss). User need to copy /etc/VRTSvcs/conf/types.cf to /etc/VRTSvcs/conf/config/types.cf, this copying may fault the resource due to missing types(like HTC) from new types.cf.RESOLUTION:Implemented new external trigger to manually update the /etc/VRTSvcs/conf/config/types.cf. Follow the post installation instructions of VRTSvcsag rpm.Patch ID: VRTSvcs-8.0.0.2400* 4111623 (Tracking ID: 4100720)SYMPTOM:GCO fails to configure for the latest RHEL/SLES platform due to an incorrect CIDR value.DESCRIPTION:GCO failed to configure as altname defined for latest RHEL/SLES kernel sends an incorrect CIDR value.RESOLUTION:Updated code to pick correct CIDR value in GCO config if altname is defined.Patch ID: VRTSvcs-8.0.0.2100* 4103077 (Tracking ID: 4103073)SYMPTOM:Security vulnerabilities present in existing version of Netsnmp.DESCRIPTION:Upgrading Netsnmp component to fix security vulnerabilitiesRESOLUTION:Upgrading Netsnmp component to fix security vulnerabilities for security patch IS 8.0U1_SP4.Patch ID: VRTSvcs-8.0.0.1800* 4084675 (Tracking ID: 4089059)SYMPTOM:File permission for gcoconfig.log is not 0600.DESCRIPTION:As default file permission was 0644 so it was allowing read access to groups and others so file permission needs to be updated.RESOLUTION:Added solution which creates file with permission 0600 so that it should be readable and writable by user.Patch ID: VRTSvcs-8.0.0.1400* 4065820 (Tracking ID: 4065819)SYMPTOM:Protocol version upgrade from Access Appliance 7.4.3.200 to 8.0 failed.DESCRIPTION:During rolling upgrade, IPM message 'MSG_CLUSTER_VERSION_UPDATE' is generated and as a part of it we do some validations for bumping up protocol. If validation succeeds then a broadcast message to bump up the cluster protocol is sent and immediately we send success message to haclus. Thus, the success message is sent before processing the actual updating Protocol version broadcast message. This process occurs for very short period. Also, after successful processing of the broadcast message, the Protocol version is properly updated in config files as well as command shows correct value.RESOLUTION:Instead of immediately returning success message, haclus CLI waits till upgrade is implemented on broadcast channel and then success message is sent.Patch ID: VRTSvxfen-8.0.0.2700* 4114265 (Tracking ID: 4122405)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL9 Update 0.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 9 Update 2(RHEL9.2) is now introduced.* 4117657 (Tracking ID: 4108561)SYMPTOM:Vxfen print keys internal utility was not working because of overrunning of array internallyDESCRIPTION:Vxfen print keys internal utility will not work if the number of keys exceed 8 will then return garbage valueOverrunning array keylist[i].key of 8 bytes at byte offset 8 using index y (which evaluates to 8)RESOLUTION:Restricted the internal loop to VXFEN_KEYLEN. Reading reservation working fine now.* 4173687 (Tracking ID: 4166666)SYMPTOM:Failed to configure Disk based fencing on rdm mapped devices from KVM host to kvm guestDESCRIPTION:While configuring read key buffer exceeding the maximum buffer size in KVM hypervisor.RESOLUTION:Reduced maximum number of keys to 1022 to support read key in KVM hypervisor.* 4176817 (Tracking ID: 4176110)SYMPTOM:vxfentsthdw failed to verify fencing disks compatibility in KVM environmentDESCRIPTION:vxfentsthdw failed to read key buffer, as exceeds the maximum buffer size in KVM hypervisor.RESOLUTION:Added new macro with less number of keys to support in kvm environment.* 4177719 (Tracking ID: 4113847)SYMPTOM:Even number of cp disks is not supported by design. This enhancement is a part of AFA wherein a faulted disk needs to be replaced as soon as the number of coordination disks is even in number and fencing is up and runningDESCRIPTION:Regular split or network partitioning must be an odd number of disks.Even number of cp support is provided with cp_count. With cp_count/2+1, fencing is not allowed to come up. Also if cp_count is not defined in vxfenmode file then by default minimum 3 cp disk are needed, otherwise vxfen does not start.RESOLUTION:In case of even number of cp disk, another disk is added. The number of cp disks is odd and fencing is thus running.Patch ID: VRTSvxfen-8.0.0.2200* 4108953 (Tracking ID: 4107779)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 7 GA kernel(4.18.0-425.3.1.el8.x86_64).RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7) is now introduced.Patch ID: VRTSvxfen-8.0.0.2100* 4102352 (Tracking ID: 4100203)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 6.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7(RHEL8.7) is now introduced.Patch ID: VRTSvxfen-8.0.0.1800* 4087166 (Tracking ID: 4087134)SYMPTOM:The error message 'Touch /var/VRTSvcs/log/vxfen/vxfen.log failed' appears after starting vxfen service, if parent directory path of vxfen.log is not present.DESCRIPTION:Typically, if parent directory path of vxfen.log is not present, the following error message appears after starting vxfen service:'Touch /var/VRTSvcs/log/vxfen/vxfen.log failed'.RESOLUTION:Create the parent directory path for the vxfen.log file globally if the path is not present.* 4088061 (Tracking ID: 4089052)SYMPTOM:On RHEL9, Node panics while running vxfenswap as a part of Online Coordination Point Replacement operation.DESCRIPTION:RHEL9 has introduced fortifying panic which gets triggered if kernel's static check finds out any buffer overflow. This check was wrongly identifying buffer overflow where strings are copied by using unions.RESOLUTION:Moved to bcopy internally for such a scenario and kernel side check skipped.Patch ID: VRTSvxfen-8.0.0.1500* 4072717 (Tracking ID: 4072335)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 6(RHEL8.6).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 5.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 6(RHEL8.6) is now introduced.Patch ID: VRTSvxfen-8.0.0.1200* 3951882 (Tracking ID: 4004248)SYMPTOM:vxfend process segfaults and coredumpedDESCRIPTION:During fencing race, sometimes vxfend crashes and generates core dumpRESOLUTION:Vxfend internally uses fork and exec to execute sub tasks. The new child process was using same file descriptors for logging purpose. This simultaneous read of same file using single file descriptor was resulting incorrect read and hence the process crash and coredump. This fix creates new file descriptor for child process to fix the crashPatch ID: VRTSvxfen-8.0.0.1100* 4064785 (Tracking ID: 4053171)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 4.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 5(RHEL8.5) is now introduced.Patch ID: VRTSgab-8.0.0.2700* 4113341 (Tracking ID: 4113340)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 8(RHEL8.8).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 7.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 8(RHEL8.8) is now introduced.Patch ID: VRTSgab-8.0.0.2200* 4108951 (Tracking ID: 4107779)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 7 GA kernel(4.18.0-425.3.1.el8.x86_64).RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7) is now introduced.Patch ID: VRTSgab-8.0.0.2100* 4100453 (Tracking ID: 4100203)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 6.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7(RHEL8.7) is now introduced.Patch ID: VRTSgab-8.0.0.1800* 4089723 (Tracking ID: 4089722)SYMPTOM:VRTSgab , VRTSamf and VRTSdbed driver does not load on RHEL and SLES platform.DESCRIPTION:Need recompilation of VRTSgab , VRTSamf and VRTSdbed with latest changes.RESOLUTION:Recompiled the VRTSgab , VRTSamf and VRTSdbed.Patch ID: VRTSgab-8.0.0.1500* 4073696 (Tracking ID: 4072335)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 6(RHEL8.6).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 5.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 6(RHEL8.6) is now introduced.Patch ID: VRTSgab-8.0.0.1100* 4064784 (Tracking ID: 4053171)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 4.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 5(RHEL8.5) is now introduced.Patch ID: VRTSllt-8.0.0.2700* 4135413 (Tracking ID: 4084657)SYMPTOM:RHEL8/7.4.1 new installation, fencing/LLT panic while using TCP over LLT.DESCRIPTION:Customer has new environment configuring Infoscale 7.4.1 with RHEL8.4, using TCP for LLT comms. The same configuration works fine with RHEL7, but system panics in RHEL8 environment.RESOLUTION:Code change done for llt binary to resolve CU panic issue.* 4135420 (Tracking ID: 3989372)SYMPTOM:When the CPU load and memory consumption is high in a VMware environment, some nodes in an InfoScale cluster may get fenced out.DESCRIPTION:Occasionally, in a VMware environment, the operating system may not schedule LLT contexts on time. Consequently, heartbeats from some of the cluster nodes may be lost, and those nodes may get fenced out. This situation typically occurs when the CPU load or the memory usage is high or when the VMDK snapshot or vMotion operations are in progress.RESOLUTION:This fix attempts to make clusters more resilient to transient issues by heartbeating using threads bound to every vCPU.* 4136152 (Tracking ID: 4124759)SYMPTOM:Panic happened with llt_ioship_recv on a server running in AWS.DESCRIPTION:In AWS environment packets can be duplicated even it is configured in UDP. As UDP it is not expected.RESOLUTION:To avoid the panic we have checked the packet is already in send queue of bucket and decided invalid/duplicate packet.* 4145248 (Tracking ID: 4139781)SYMPTOM:System panics occasionally in LLT stack where LLT over ether enabled.DESCRIPTION:LLT allocates skb memory from own cache for those >4k messages and sets a field of skb_shared_info pointing to a LLT function, later uses the field to determine if skb is allocated from own cache. When receiving a packet, OS also allocates skb from system cache and doesn't reset the field, then passes skb to LLT. Occasionally the stale pointer in memory can mislead LLT to think a skb is from own cache and uses LLT free api by mistake.RESOLUTION:LLT uses a hash table to record skb allocated from own cache and doesn't set the field of skb_shared_info.* 4156794 (Tracking ID: 4135825)SYMPTOM:Once root file system is full during llt start, llt module failing to load forever.DESCRIPTION:Disk is full, user rebooted system or restart the product. In this case while LLT loading, it deletes llt links and tried creates new one using link names and "/bin/ln -f -s". As disk is full, it unable to create the links. Even after making space, it failed to create the links as those are deleted. So failed to load LLT module.RESOLUTION:If existing links are not present, added the logic to get name of file names to create new links.* 4156815 (Tracking ID: 4087543)SYMPTOM:Node panic observed at llt_rdma_process_ack+189DESCRIPTION:LLT in llt_rdma_process_ack() function, trying to access the header mblk becomes invalid, and sends an unnecessary acknowledgement from the network/rdma layer. When ib_post_send() function fails, OS returns an error, and LLT takes care of this error by sending the packet through non-rdma channel. Even when the OS has returned the error, that packet is still sent down and LLT get an acknowledgement. LLT thus receives two acknowledgements for the same buffer, one which rdma layer sends although it report errors while sending, and other that LLT simulates (by design) after delivering the packet through non-rdma channel.The first acknowledgement context frees up the buffer and so when LLT again calls same function (llt_rdma_process_ack() ) for the same buffer, as the buffer has already been freed, LLT hits panic in that function.RESOLUTION:Added a check that stops buffer being freed up twice and hence does not panic node at llt_rdma_process_ack+189.Patch ID: VRTSllt-8.0.0.2600* 4135413 (Tracking ID: 4084657)SYMPTOM:RHEL8/7.4.1 new installation, fencing/LLT panic while using TCP over LLT.DESCRIPTION:Customer has new environment configuring Infoscale 7.4.1 with RHEL8.4, using TCP for LLT comms. The same configuration works fine with RHEL7, but system panics in RHEL8 environment.RESOLUTION:Code change done for llt binary to resolve CU panic issue.* 4135420 (Tracking ID: 3989372)SYMPTOM:When the CPU load and memory consumption is high in a VMware environment, some nodes in an InfoScale cluster may get fenced out.DESCRIPTION:Occasionally, in a VMware environment, the operating system may not schedule LLT contexts on time. Consequently, heartbeats from some of the cluster nodes may be lost, and those nodes may get fenced out. This situation typically occurs when the CPU load or the memory usage is high or when the VMDK snapshot or vMotion operations are in progress.RESOLUTION:This fix attempts to make clusters more resilient to transient issues by heartbeating using threads bound to every vCPU.* 4136152 (Tracking ID: 4124759)SYMPTOM:Panic happened with llt_ioship_recv on a server running in AWS.DESCRIPTION:In AWS environment packets can be duplicated even it is configured in UDP. As UDP it is not expected.RESOLUTION:To avoid the panic we have checked the packet is already in send queue of bucket and decided invalid/duplicate packet.* 4145248 (Tracking ID: 4139781)SYMPTOM:System panics occasionally in LLT stack where LLT over ether enabled.DESCRIPTION:LLT allocates skb memory from own cache for those >4k messages and sets a field of skb_shared_info pointing to a LLT function, later uses the field to determine if skb is allocated from own cache. When receiving a packet, OS also allocates skb from system cache and doesn't reset the field, then passes skb to LLT. Occasionally the stale pointer in memory can mislead LLT to think a skb is from own cache and uses LLT free api by mistake.RESOLUTION:LLT uses a hash table to record skb allocated from own cache and doesn't set the field of skb_shared_info.* 4156794 (Tracking ID: 4135825)SYMPTOM:Once root file system is full during llt start, llt module failing to load forever.DESCRIPTION:Disk is full, user rebooted system or restart the product. In this case while LLT loading, it deletes llt links and tried creates new one using link names and "/bin/ln -f -s". As disk is full, it unable to create the links. Even after making space, it failed to create the links as those are deleted. So failed to load LLT module.RESOLUTION:If existing links are not present, added the logic to get name of file names to create new links.* 4156815 (Tracking ID: 4087543)SYMPTOM:Node panic observed at llt_rdma_process_ack+189DESCRIPTION:LLT in llt_rdma_process_ack() function, trying to access the header mblk becomes invalid, and sends an unnecessary acknowledgement from the network/rdma layer. When ib_post_send() function fails, OS returns an error, and LLT takes care of this error by sending the packet through non-rdma channel. Even when the OS has returned the error, that packet is still sent down and LLT get an acknowledgement. LLT thus receives two acknowledgements for the same buffer, one which rdma layer sends although it report errors while sending, and other that LLT simulates (by design) after delivering the packet through non-rdma channel.The first acknowledgement context frees up the buffer and so when LLT again calls same function (llt_rdma_process_ack() ) for the same buffer, as the buffer has already been freed, LLT hits panic in that function.RESOLUTION:Added a check that stops buffer being freed up twice and hence does not panic node at llt_rdma_process_ack+189.* 4156824 (Tracking ID: 4138779)SYMPTOM:Veritas Infoscale Availability modules will not work (function) on Red Hat Enterprise Linux 8 Update 9(RHEL8.9).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 9(RHEL8.9).RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 9(RHEL8.9) is now introduced.Patch ID: VRTSllt-8.0.0.2400* 4116421 (Tracking ID: 4113340)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 8(RHEL8.8).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 7.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 8(RHEL8.8) is now introduced.Patch ID: VRTSllt-8.0.0.2200* 4108947 (Tracking ID: 4107779)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 7 GA kernel(4.18.0-425.3.1.el8.x86_64).RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7) is now introduced.Patch ID: VRTSllt-8.0.0.2100* 4101232 (Tracking ID: 4100203)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 6.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7(RHEL8.7) is now introduced.Patch ID: VRTSllt-8.0.0.1800* 4061158 (Tracking ID: 4061156)SYMPTOM:IO error on /sys/kernel/slab folderDESCRIPTION:After loading LLT module, LS command throws IO error on /sys/kernel/slab folderRESOLUTION:IO error on /sys/kernel/slab folder is now fixed after loading LLT module* 4079637 (Tracking ID: 4079636)SYMPTOM:Kernel is getting panicked with null pointer dereference in llt_dump_mblk when LLT is configured over IPsecDESCRIPTION:LLT uses skb's sp pointer to chain socekt buffers internally. When LLT is configured over IPsec, llt will receive skb's with sp pointer from ip layer. These skbs were wrongly identified by llt as chained skbs. Now we are resetting the sp pointer field before re-using for interanl chaining.RESOLUTION:No panic observed after applying this patch.* 4079662 (Tracking ID: 3981917)SYMPTOM:LLT UDP multiport was previously supported only on 9000 MTU networks.DESCRIPTION:Previously LLT UDP multiport configuration required network links to have 9000 MTU. We have enhanced UDP multiport code, so that now this LLT feature can be configured/run on 1500 MTU links as well.RESOLUTION:LLT UDP multiport can be configured on 1500 MTU based networks* 4080630 (Tracking ID: 4046953)SYMPTOM:During LLT configuration, messages related to 9000 MTU are getting printed as error.DESCRIPTION:On Azure error messages related to 9000 MTU are getting logged. These message indicates that to have optimal performance , use 9000 MTU based networks . These are not actual errors but suggestion.RESOLUTION:Since customer is going to use it on Azure where 9000 MTU is not supported, hence removed these messages to avoid confusion.Patch ID: VRTSllt-8.0.0.1500* 4073695 (Tracking ID: 4072335)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 6(RHEL8.6).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 5.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 6(RHEL8.6) is now introduced.Patch ID: VRTSllt-8.0.0.1200* 4066063 (Tracking ID: 4066062)SYMPTOM:Node panicDESCRIPTION:Node panic observed in llt udp multiport configuration with vx ioship stack.RESOLUTION:When llt receives an acknowledgement, it tries to free the packet and corresponding client frags blindly without checking the client status. If the client is unregistered, then the free functions of the frags will be invalid and hence should not be called.* 4066667 (Tracking ID: 4040261)SYMPTOM:During LLT configuration, if set-verbose is set to 1 in /etc/llttab, an lltconfig core dump is observed.DESCRIPTION:Some log messages may have IDs like 00000. When such logs are encountered, it may lead to a core dump by the lltconfig process.RESOLUTION:VCS is updated to use appropriate message IDs for logs so that such issues do not occur.Patch ID: VRTSllt-8.0.0.1100* 4064783 (Tracking ID: 4053171)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 4.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 5(RHEL8.5) is now introduced.Patch ID: VRTSvcsea-8.0.0.2500* 4118769 (Tracking ID: 4073508)SYMPTOM:Oracle virtual fire-drill is failing due to Oracle password file location changes from Oracle version 21c.DESCRIPTION:Oracle password file has been moved to $ORACLE_BASE/dbs from Oracle version 21c.RESOLUTION:Environment variables are used for pointing the updated path for the password file.It is mandatory from Oracle 21c and later versions for a client to configure .env file path in EnvFile attribute. This file must have ORACLE_BASE path added to work Oracle virtual fire-drill feature. Sample EnvFile content with ORACLE_BASE path for Oracle 21c [root@inaqalnx013 Oracle]# cat /opt/VRTSagents/ha/bin/Oracle/envfile ORACLE_BASE="/u02/app/oracle/product/21.0.0/dbhome_1/"; export ORACLE_BASE; Sample attribute value EnvFile = "/opt/VRTSagents/ha/bin/Oracle/envfile"Patch ID: VRTSvcsea-8.0.0.1800* 4030767 (Tracking ID: 4088595)SYMPTOM:hapdbmigrate utility fails to online the oracle service group due to a timing issue.DESCRIPTION:hapdbmigrate utility fails to online the oracle service group due to a timing issue. example: ./hapdbmigrate -pdbres pdb1_res -cdbres cdb2_res -XMLdirectory /oracle_xml Cluster prechecks and validation Done Taking PDB resource [pdb1_res] offline Done Modification of cluster configuration Done VCS ERROR V-16-41-39 Group [CDB2_grp] is not ONLINE after 300 seconds on %vcs_node% VCS ERROR V-16-41-41 Group [CDB2_grp] is not ONLINE on some nodes in the cluster Bringing PDB resource [pdb1_res] online on CDB resource [cdb2_res]Done For further details, see '/var/VRTSvcs/log/hapdbmigrate.log'RESOLUTION:hapdbmigrate utility modified to ensure enough time elapses between probe of PDB resource and online of CDB group.* 4079559 (Tracking ID: 4064917)SYMPTOM:Oracle agent fails to generate ora_api (which is used for Intentional Offline functionality of Oracle agent) using build_oraapi.sh script for Oracle 21c.DESCRIPTION:The build_oraapi.sh script could not connect to library named libdbtools21.a, as on the new Oracle 21c environment generic library is present i.e. '$ORACLE_HOME/rdbms/lib/libdbtools.a'.RESOLUTION:This script on Oracle 21c database environment will pick generic library and on older database environments it will pick Database version specific library.Patch ID: VRTSvcsag-8.0.0.2700* 4121275 (Tracking ID: 4121270)SYMPTOM:EBSvol agent error in attach disk : RHEL 7.9 + Infoscale 8.0 on AWS instance type c6i.large with NVME devices.DESCRIPTION:After attaching volume to instance its taking some time to update its device mapping in system. Due to which if we run lsblk -d -o +SERIAL immediately after attaching volume then its not showing that volume details in output. Due to which $native_device was getting blank/uninitialized. So, we need to wait for some time to get device mapping updated in system.RESOLUTION:We have added logic to retry once for same command after some interval if in first run we didnt find expected volume device mapping. And now its properly updating attribute NativeDevice.* 4122004 (Tracking ID: 4122001)SYMPTOM:NIC resource remain online after unplug network cable on ESXi server.DESCRIPTION:Previously MII checking network statistics/ping test but now it's directly marking NIC state ONLINE by checking NIC status in operstate. Now there is no any ping check before that. If it's fail to detect operstate file then only it's going to check for PING test. But on ESXi server environment NIC is already marked ONLINE as operstate file is available with state UP & carrier bit is set. So, even if "NetworkHosts" is not reachable then also it's marking NIC resource ONLINE.RESOLUTION:The NIC agent's already having "PingOptimize" attribute. We introduced new value (2) for "PingOptimize" attribute to make decision of perform PING test. If "PingOptimize = 2" then only it will do PING test or else it will work as per previous design.* 4127323 (Tracking ID: 4127320)SYMPTOM:The ProcessOnOnly agent fails to bring online a resource when a user shell is set to /sbin/nologin.DESCRIPTION:The agent fails to bring online a resource when the shell for the user is set to /sbin/nologin.RESOLUTION:The ProcessOnOnly agent is enhanced to support the /sbin/nologin shell. If the shell is set to /sbin/nologin, the agent uses /bin/bash as shell to start the process.* 4161344 (Tracking ID: 4152812)SYMPTOM:AWS EBSVol agent takes long time to perform online and offline operations on resources.DESCRIPTION:When a large number of AWS EBSVol resources are configured, it takes a long time to perform online and offline operations on these resources. EBSVol is a single threaded agent and hence prevents parallel execution of attach and detach EBS volume commands.RESOLUTION:To resolvethe issue, the default value of 'NumThreads' attribute of EBSVol agent is modified from 1 to 10 and the agent is enhanced to use the locking mechanism to avoid conflicting resource configuration. This results in enhanced response time for parallel execution of attach and detach commands. Also, the default value of MonitorTimeout attribute is modified from 60 to 120. This avoids timeout of monitor entry point when response of AWS CLI/server is unexpectedly slow.* 4161350 (Tracking ID: 4152815)SYMPTOM:AWS EBS Volume in-use with other AWS instance is getting used by cluster nodes through AWS EBSVol agent.DESCRIPTION:AWS EBS Volume which is attached to AWS instance that is not part of cluster is getting attach to the node of cluster during online event. Instead, Unable to detach volume' message should be logged in log file as volume is already in use by another AWS instance in AWS EBSVol agent.RESOLUTION:AWS EBSVol agent is enhanced to avoid attachment of in-use EBS volumes whose instances are not part of cluster.* 4162952 (Tracking ID: 4142040)SYMPTOM:While upgrading the VRTSvcsag rpm package, the '/etc/VRTSvcs.conf/config/types.cf' file on Veritas Cluster Server(VCS) might be incorrectly updated.DESCRIPTION:While upgrading the VRTSvcsag rpm package, the '/etc/VRTSvcs.conf/config/types.cf' file on Veritas Cluster Server(VCS) might be incorrectly updated.During some instances, the user might be informed to manually copy '/etc/VRTSvcs/conf/types.cf' to the existing '/etc/VRTSvcs/conf/config/types.cf' file. Need to remove the message "Implement /etc/VRTSvcs/conf/types.cf to utilize resource type updates" when updating the VRTSvcsag rpm.RESOLUTION:To ensure that '/etc/VRTSvcs/conf/config/types.cf file' is updated correctly following VRTSvcsag updates, the script user_trigger_update_types can be manually triggered by the user. The following message displays:Leaving existing /etc/VRTSvcs/conf/config/types.cf configuration file unmodifiedCopy /opt/VRTSvcs/bin/sample_triggers/VRTSvcs/user_trigger_update_types to /opt/VRTSvcs/bin/triggersTo manually update the types.cf, execute command "hatrigger -user_trigger_update_types 0* 4163231 (Tracking ID: 4152700)SYMPTOM:When Private DNS Zone resource ID is passed, the AzureDNSZone Agent returns an error saying that the resource cannot be found.DESCRIPTION:Azure Private DNS Zone with AzureDNSZone Agent is not supported.RESOLUTION:The Azure Private DNS Zone is supported by the AzureDNSZone Agent by installing the Azure library for Private DNS Zone(azure-mgmt-privatedns). This library has functions that can be utilized by for Private DNS zone operations. The resource ID is differentiated based on the Public and the Private DNS zones and the corrective actions are taken accordingly. For DNS zones, the resource ID differs between Public and Private DNS zones. The resource ID can be parsed, and the resource type can be checked to determine whether it is a Public or Private DNS zone.* 4163234 (Tracking ID: 4152886)SYMPTOM:AWSIP agent fails to bring OverlayIP resources online and offline on the instances in a VPC that is shared across multiple AWS accounts.DESCRIPTION:When VPC is shared across multiple AWS accounts, route table associated with the subnets is exclusively owned by the owner account. AWS restricts the modification in the route table from any other account. When AWSIP agent tries to bring OverlayIP resource online on the instance owned by a different account, the account may not have privileges to update the route table. In such cases, AWSIP agent fails to edit the route table, and fails to bring OverlayIP resource online and offline.RESOLUTION:To support cross-account deployment, assign appropriate privileges on the shared resources. Create an AWS profile to grant permissions to update the route table of VPC through different nodes belonging to different AWS accounts. This profile is used by the AWSIP agent to update route tables.A new attribute "Profile" is introduced in the AWSIP agent. Configure this attribute with using the newly created profile.Note: Adding "ec2:DescribeVpcs" permission is mandatory to use this patch as this is being used to decide if VPC is shared or not. Refer to the following technote for more details. https://sort.veritas.com/DocPortal/pdf/infoscale_802u2_awsiptechnote* 4165268 (Tracking ID: 4162658)SYMPTOM:LVMVolumeGroup resource fails to offline/clean in cloud environment after path failure.DESCRIPTION:If disk is detached unable to failover LVMVolumeGroup.RESOLUTION:Implementation of PanicSystemOnVGLoss attribute.0 - Default value and behaviour, does not failover (not halting the system).1 Halt the system if deactivation of volume group fails.2 - Do not halt the system. Allow failover (Note risk of data corruption).* 4169950 (Tracking ID: 4169794)SYMPTOM:NIC resource FAULTS when we there is no IP and attributes like NetworkHosts,PingTest configured for any interface/aggregationDESCRIPTION:For network aggregations or interfaces which have no IP set, but are to be monitored will fail with checks like ipadm, PingTest. To monitor a device state at the data link layer, add a new attribute 'LinkTest' to monitor a n/w device by using 'dladm' and skip the checks on IP(like ipadm, ping test). So, even if 'NetworkHosts' or 'IP' is not set the resource will not fault and will report ONLINE based on the state from dladm.RESOLUTION:Add a new attribute 'LinkTest' to monitor a n/w device for which no IP is configured, so that resource can be monitored both by dladm at data link layer, and/or by ipadm, pin tests at the network layer as per the requirement.Patch ID: VRTSvcsag-8.0.0.2500* 4118318 (Tracking ID: 4113151)SYMPTOM:Dependent DiskGroupAgent fails to get its resource online due to disk group import failure.DESCRIPTION:VMwareDisksAgent reports its resource online just after VMware disk is attached to virutal machine, if dependent DiskGroup resource starts to online at the moment it must fail because VMware disk is not yet present into vxdmp database due to VxVM transaction latency. Customer used to add retry times to work around this problem but cannot apply the same to every environment.RESOLUTION:Added a finite period of wait for VMware disk is present into vxdmp database before online is complete.* 4118448 (Tracking ID: 4075950)SYMPTOM:When IPv6 VIP switches from node1 to node2 in a cluster,a longer time is taken to update its neighboring information and traffic to reach node2 which is on the reassigned address.DESCRIPTION:After the Service group switches from node1 to node2, the IPv6 VIP is not reachable from the network switch. The mac address changes after the node switch, but the network is not updated. Similar to IPv4 VIP by gracious ARP, in case of IPV6 VIP switch from node1 to node2; the network must be updated for the mac address change.RESOLUTION:The network devices which communicate with the VIP are not able to establish a connection with the VIP. To connect with the VIP, the VIP is pinged from the switch or from the cluster nodes 'ip -6 neighbor flush all' command is run. Neighbour flush logic is added to IP/MultiNIC agents so that the changed mac id during floating VIP switchover is updated in the network.* 4118455 (Tracking ID: 4118454)SYMPTOM:when root user login shell is set to /sbin/nologin in /etc/passwd file, Process agent resource fails to come online.DESCRIPTION:From the engine_A.log, the below errors were logged:2023/05/31 11:34:52 VCS NOTICE V-16-10031-20704 Process:Process:imf_getnotification:Received notification for vxamf-group sendmail2023/05/31 11:35:38 VCS ERROR V-16-10031-9502 Process:sendmail:online:Could not online the resource, make sure user-name is correct.2023/05/31 11:35:39 VCS INFO V-16-2-13716 Thread(140147853162240) Resource(sendmail): Output of the completed operation (online)==============================================This account is currently not available.==============================================RESOLUTION:The Process agent is enhanced to support nologin shell for root user. If user shell is set to /sbin/nologin, the agent starts a process using /bin/bash shell.* 4118767 (Tracking ID: 4094539)SYMPTOM:The MonitorProcesses argument in the resource ArgListValues being passed to the agent (bundled ApplicationAgent) is incorrectly removing an extra needed space from the following process, as found via the recommended CLI process test.DESCRIPTION:In the ArgListValues under MonitorProcesses with the extra space it even shows up when displaying the resource.RESOLUTION:For the monitored process (not program) only remove leading and trailing spaces. Do not remove extra spaces between words.Patch ID: VRTSvcsag-8.0.0.2400* 4113057 (Tracking ID: 4113056)SYMPTOM:ReuseMntPt is not honored when the same mountpoint is used for two resources with different FSType.DESCRIPTION:The reuseMntPt attribute is set to 1 if the same mount point needs to be specified in more than one mount resource but when both the resource has different FSType, The state of the offline resource shows as offline|unknown instead of offline. No errors on the online resource.RESOLUTION:The Required code changes have been done to indicate the correct state of the resource.Patch ID: VRTSvcsag-8.0.0.1800* 4030767 (Tracking ID: 4088595)SYMPTOM:hapdbmigrate utility fails to online the oracle service group due to a timing issue.DESCRIPTION:hapdbmigrate utility fails to online the oracle service group due to a timing issue.example:./hapdbmigrate -pdbres pdb1_res -cdbres cdb2_res -XMLdirectory /oracle_xmlCluster prechecks and validation DoneTaking PDB resource [pdb1_res] offline DoneModification of cluster configuration DoneVCS ERROR V-16-41-39 Group [CDB2_grp] is not ONLINE after 300 seconds on %vcs_node%VCS ERROR V-16-41-41 Group [CDB2_grp] is not ONLINE on some nodes in the clusterBringing PDB resource [pdb1_res] online on CDB resource [cdb2_res]DoneFor further details, see '/var/VRTSvcs/log/hapdbmigrate.log'RESOLUTION:hapdbmigrate utility modified to ensure enough time elapses between probe of PDB resource and online of CDB group.* 4058802 (Tracking ID: 4073842)SYMPTOM:Oracle 21c is not supported on earlier product versions.DESCRIPTION:Implemented Oracle 21c support with Storage Foundation for Databases.RESOLUTION:Changes are done to support Oracle 21c with Storage Foundation for Databases.* 4079372 (Tracking ID: 4073842)SYMPTOM:Oracle 21c is not supported on earlier product versions.DESCRIPTION:Implemented Oracle 21c support with Storage Foundation for Databases.RESOLUTION:Changes are done to support Oracle 21c with Storage Foundation for Databases.* 4079559 (Tracking ID: 4064917)SYMPTOM:Oracle agent fails to generate ora_api (which is used for Intentional Offline functionality of Oracle agent) using build_oraapi.sh script for Oracle 21c.DESCRIPTION:The build_oraapi.sh script could not connect to library named libdbtools21.a, as on the new Oracle 21c environment generic library is present i.e. '$ORACLE_HOME/rdbms/lib/libdbtools.a'.RESOLUTION:This script on Oracle 21c database environment will pick generic library and on older database environments it will pick Database version specific library.* 4081774 (Tracking ID: 4083099)SYMPTOM:When OverlayIP is configured AzureIP resource offline operation fails.DESCRIPTION:AzureIP resource fails to go offline when OverlayIP is configured because Azure API routes.delete part of azure-mgmt-network module has been deprecated.RESOLUTION:A new API routes.begin_delete is introduced as suggested by Azure in the Azure agent.Patch ID: VRTScps-8.0.0.2700* 4152884 (Tracking ID: 4152882)SYMPTOM:Intermittently losing access to the CP serversDESCRIPTION:Since we write logs into every log files(vxcpserve_[A|B|C].log ) at most till maxlen, but if it goes beyond that length, a new file will be opened and the old one will be closed. At this stack, fptr uses the old pointer, resulting in fwrite() to a closed FILE ptr.RESOLUTION:Opened a new file before assignment of fptr so that it will point to a correct FILE ptr.Patch ID: VRTScps-8.0.0.1900* 4091306 (Tracking ID: 4088158)SYMPTOM:Security vulnerabilities exists Sqlite third-party components used by VCS.DESCRIPTION:VCS uses the Sqlite third-party components in which some security vulnerability exist.RESOLUTION:VCS is updated to use newer versions of Sqlite third-party components in which the security vulnerabilities have been addressed.Patch ID: VRTScps-8.0.0.1800* 4073050 (Tracking ID: 4018218)SYMPTOM:Secure communication between a CP Server and a CP Client cannot be established using TLSv1.2DESCRIPTION:Secure communication between a CP Server and a CP Client cannot be established using TLSv1.2.RESOLUTION:This hotfix updates the VRTScps module so that InfoScale CP Client can establish secure communication with a CP server using TLSv1.2. However, to enable TLSv1.2 communication between the CP client and CP server after installing this hotfix, you must perform the following steps:To configure TLSv1.2 for CP server1. Stop the process resource that has pathname="/opt/VRTScps/bin/vxcpserv" # hares -offline <vxcpserv> -sys <sysname> 2. Check that the vxcpserv daemon is stopped using the following command: # ps -eaf | grep "/opt/VRTScps/bin/vxcpserv"3. When the vxcpserv daemon is stopped, edit the "/etc/vxcps_ssl.properties" file and make the following changes: a. Remove or comment the entry: openSSL.server.requireTLSv1 = true b. Add a new entry: openSSL.server.requireTLSv1.2 = true4. Start the process resource that has pathname="/opt/VRTScps/bin/vxcpserv" # hares -offline <vxcpserv> -sys <sysname>To configure TLSv1.2 for CP ClientEdit the "/etc/vxcps_ssl.properties" file and make the following changes: a. Remove or comment the entry: openSSL.server.requireTLSv1 = true b. Add a new entry: openSSL.server.requireTLSv1.2 = truePatch ID: VRTScps-8.0.0.1200* 4066225 (Tracking ID: 4056666)SYMPTOM:The Error writing to database message may appear in syslogs intermittently on InfoScale CP servers.DESCRIPTION:Typically, when a coordination point server (CP server) is shared among multiple InfoScale clusters, the following messages may intermittently appear in syslogs:CPS CRITICAL V-97-1400-501 Error writing to database! :database is locked.These messages appear in the context of the CP server protocol handshake between the clients and the server.RESOLUTION:The CP server is updated so that, in addition to its other database write operations, all the ones for the CP server protocol handshake action are also synchronized.Patch ID: VRTSamf-8.0.0.2700* 4145249 (Tracking ID: 4136003)SYMPTOM:A cluster node panics when VCS enabled AMF module that monitors process on/off.DESCRIPTION:A cluster node panics which indicates that an issue with the AMF module overruns into user space buffer during it is analyzing an argument of 8K size. The AMF module cannot load that length of data into internal buffer, eventually misleading access into user buffer which is not allowed when kernel SMAP in effect.RESOLUTION:The AMF module is constrained to ignore an argument of 8K or bigger size to avoid internal buffer overrun.* 4162992 (Tracking ID: 4161644)SYMPTOM:System panics when VCS enabled AMF module.DESCRIPTION:System panics that indicates after amf_prexec_hook extracted an argument longer than 4k which spansover two pages it is reading the 3rd page, that is illegal because all arguments are loaded in two pages.RESOLUTION:AMF continue to extract arguments from internal buffer before moving to next page.Patch ID: VRTSamf-8.0.0.2400* 4116411 (Tracking ID: 4113340)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 8(RHEL8.8).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 7.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 8(RHEL8.8) is now introduced.Patch ID: VRTSamf-8.0.0.2200* 4108952 (Tracking ID: 4107779)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 7 GA kernel(4.18.0-425.3.1.el8.x86_64).RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7 on latest minor kernel 4.18.0-425.10.1.el8_7.x86_64(RHEL8.7) is now introduced.Patch ID: VRTSamf-8.0.0.2100* 4101233 (Tracking ID: 4100203)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 7(RHEL8.7).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 6.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 7(RHEL8.7) is now introduced.Patch ID: VRTSamf-8.0.0.1800* 4089724 (Tracking ID: 4089722)SYMPTOM:VRTSgab , VRTSamf and VRTSdbed driver does not load on RHEL and SLES platform.DESCRIPTION:Need recompilation of VRTSgab , VRTSamf and VRTSdbed with latest changes.RESOLUTION:Recompiled the VRTSgab , VRTSamf and VRTSdbed.Patch ID: VRTSamf-8.0.0.1500* 4072752 (Tracking ID: 4072335)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 6(RHEL8.6).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 5.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 6(RHEL8.6) is now introduced.Patch ID: VRTSamf-8.0.0.1100* 4064788 (Tracking ID: 4053171)SYMPTOM:Veritas Infoscale Availability does not support Red Hat Enterprise Linux 8 Update 5(RHEL8.5).DESCRIPTION:Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL8 Update 4.RESOLUTION:Veritas Infoscale Availability support for Red Hat Enterprise Linux 8 Update 5(RHEL8.5) is now introduced.Patch ID: VRTSvxvm-8.0.0.2900* 4058590 (Tracking ID: 4058867)SYMPTOM:Old VxVM rpm fails to load on RHEL8.7 minor kernel 4.18.0-425.10.1.el8_7.x86_64DESCRIPTION:RedHat did some critical changes in latest kernel which causing soft-lockup issue to VxVM kernel modules while installation.RESOLUTION:As suggested by RedHat (https://access.redhat.com/solutions/6985596) VxVM modules compiled with RHEL 8.7 minor kernel.* 4067237 (Tracking ID: 4058894)SYMPTOM:After package installation and reboot , messages regarding udev rules for ignore_device are observed in /var/log/messages .systemd-udevd[774]: /etc/udev/rules.d/40-VxVM.rules:25 Invalid value for OPTIONS key, ignoring: 'ignore_device'DESCRIPTION:From SLES15 Sp3 onwards , ignore_device is deprecated from udev rules and is not available for use anymore . Hence these messages are observed in system logs .RESOLUTION:Required changes have been done to handle this defect.* 4070100 (Tracking ID: 3159650)SYMPTOM:vxtune did not support vol_vvr_use_nat.DESCRIPTION:Platform specific methods were required to set vol_vvr_use_nat tunable, as its support for vxtune command was not present.RESOLUTION:Added vol_vvr_use_nat support for vxtune command.* 4103795 (Tracking ID: 4011780)SYMPTOM:This is new array and we need to add support for EMC PowerStore plus PP.DESCRIPTION:EMC PowerStore is new array and current ASL doesn't support it. So it will not be claimed with current ASL. This array support has been now added in the current ASL.RESOLUTION:Code changes to support EMC PowerStore plus PP have been done.* 4104264 (Tracking ID: 4098391)SYMPTOM:Kernel panic is observed with following stack:#6 [ffffa479c21cf6f0] page_fault at ffffffffb240130e [exception RIP: bfq_bio_bfqg+37] RIP: ffffffffb1e78135 RSP: ffffa479c21cf7a0 RFLAGS: 00010002 RAX: 000000000000001f RBX: 0000000000000000 RCX: ffffa479c21cf860 RDX: ffff8bd779775000 RSI: ffff8bd795b2fa00 RDI: ffff8bd795b2fa00 RBP: ffff8bd78f136000 R8: 0000000000000000 R9: ffff8bd793a5b800 R10: ffffa479c21cf828 R11: 0000000000001000 R12: ffff8bd7796b6e60 R13: ffff8bd78f136000 R14: ffff8bd795b2fa00 R15: ffff8bd7946ad0bc ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018#7 [ffffa479c21cf7b0] bfq_bic_update_cgroup at ffffffffb1e78458#8 [ffffa479c21cf7e8] bfq_bio_merge at ffffffffb1e6f47f#9 [ffffa479c21cf840] blk_mq_submit_bio at ffffffffb1e48c09#10 [ffffa479c21cf8c8] submit_bio_noacct at ffffffffb1e3c7e3#11 [ffffa479c21cf958] submit_bio at ffffffffb1e3c87b#12 [ffffa479c21cf9a8] submit_bh_wbc at ffffffffb1d2536a#13 [ffffa479c21cf9e0] block_read_full_page at ffffffffb1d27ac1#14 [ffffa479c21cfa90] do_read_cache_page at ffffffffb1c2f7e5#15 [ffffa479c21cfb48] read_part_sector at ffffffffb1e546b5#16 [ffffa479c21cfb60] read_lba at ffffffffb1e595d2#17 [ffffa479c21cfba8] efi_partition at ffffffffb1e59f4d#18 [ffffa479c21cfcb8] blk_add_partitions at ffffffffb1e54377#19 [ffffa479c21cfcf8] bdev_disk_changed at ffffffffb1d2a8fa#20 [ffffa479c21cfd30] __blkdev_get at ffffffffb1d2c16c#21 [ffffa479c21cfda0] blkdev_get at ffffffffb1d2c2b4#22 [ffffa479c21cfdb8] __device_add_disk at ffffffffb1e5107e#23 [ffffa479c21cfe20] dmp_register_disk at ffffffffc0e68ae7 [vxdmp]#24 [ffffa479c21cfe50] dmp_reconfigure_db at ffffffffc0e8d8bd [vxdmp]#25 [ffffa479c21cfe80] dmpioctl at ffffffffc0e75cd5 [vxdmp]#26 [ffffa479c21cfe90] dmp_ioctl at ffffffffc0e9d469 [vxdmp]#27 [ffffa479c21cfea8] blkdev_ioctl at ffffffffb1e4ed19#28 [ffffa479c21cfef0] block_ioctl at ffffffffb1d2a719#29 [ffffa479c21cfef8] ksys_ioctl at ffffffffb1cfb262#30 [ffffa479c21cff30] __x64_sys_ioctl at ffffffffb1cfb296#31 [ffffa479c21cff38] do_syscall_64 at ffffffffb1a0538b#32 [ffffa479c21cff50] entry_SYSCALL_64_after_hwframe at ffffffffb240008cDESCRIPTION:VxVM causes kernel panic because of null pointer dereference in kernel code when BFQ disk io scheduler is used. This is observed on SLES15 SP3 minor kernel &gt;= 5.3.18-150300.59.68.1 and SLES15 SP4 minor kernel &gt;= 5.14.21-150400.24.11.1RESOLUTION:Code changes have been done to fix this issue in IS-8.0 and IS-8.0.2.* 4104757 (Tracking ID: 4055159)SYMPTOM:vxdisk list showing incorrect value of LUN_SIZE for nvme disksDESCRIPTION:vxdisk list showing incorrect value of LUN_SIZE for nvme disks.RESOLUTION:Code changes have been done to show correct LUN_SIZE for nvme devices.* 4104984 (Tracking ID: 4095163)SYMPTOM:System panic with below stack: #6 [] invalid_op at [exception RIP: __slab_free+414] #7 [] kfree at #8 [] vol_ru_free_update at [vxio] #9 [] vol_ru_free_updateq at [vxio]#10 [] vol_rv_write2_done at [vxio]#11 [] voliod_iohandle at [vxio]#12 [] voliod_loop at [vxio]DESCRIPTION:The update gets freed as a part of VVR recovery. At the same time, this update also gets freed in VVR second phase of write. Hence there is a race in freeing the updates and caused the system panic.RESOLUTION:Code changes have been made to avoid* 4105598 (Tracking ID: 4107801)SYMPTOM:/dev/vx/.dmp hardware path entries are not getting created on SLES15SP3 onwards.DESCRIPTION:vxpath-links is responsible for creating the the hardware paths under /dev/vx/.dmp .This script get invokes from: /lib/udev/vxpath_links. The "/lib/udev" folder is not present in SLES15SP3.This folder is explicitly removed from SLES15SP3 onwards and it is expected to create Veritas specific scripts/libraries from vendor specific folder.RESOLUTION:Code changes have been made to invoke "/etc/vx/vxpath-links" instead of "/lib/udev/vxpath-links".* 4108322 (Tracking ID: 4107083)SYMPTOM:In case of EMC BCV NR LUNs, vxconfigd taking a long time to start post reboot.DESCRIPTION:This issue is introduced due to BCV NR LUNs going into error state and the the scci inquiry succeed but the disk retry takes time as it goes into loop for each disk. This is a corner case and was not handled for BCV NR LUNs.RESOLUTION:Necessary code changes are done incase of BCV NR LUNs when scci inquiry succeeds, we mark disk as failed so that the vxconfigd boots quickly.* 4108392 (Tracking ID: 4107802)SYMPTOM:vxdmp fails to load and system hangs.DESCRIPTION:This issue occurs due to changes in the RHEL8.7 minor kernel and incorrect module is calculated for best-fit.RESOLUTION:Modified existing modinst-vxvm script to calculate correct best-fit module.* 4109554 (Tracking ID: 4105953)SYMPTOM:System panic with below stack in CVR environment. #9 [] page_fault at [exception RIP: vol_ru_check_update_done+183]#10 [] vol_rv_write2_done at [vxio]#11 [] voliod_iohandle at [vxio]#12 [] voliod_loop at [vxio]#13 [] kthread atDESCRIPTION:In CVR environment, when IO is issued in writeack sync mode we ack to application when datavolwrite is done on either log client or logowner depending on where IO is issued on. it could happen that VVR freed the metadata I/O update after SRL write is done incase of writeack sync mode, but later after freeing the update, its accessed again and hence we end up in hitting NULL ptr deference.RESOLUTION:Code changes have been made to avoid the accessing NULL pointer.* 4110398 (Tracking ID: 4106254)SYMPTOM:Nodes crashed in shared-nothing (Flexible Shared Storage) environment if node reboot followed by NVME disk failure is executedDESCRIPTION:If congested functions are registered in Linux driver, those are called to check if next set of IOs can be issued on devices and device can handle those.In this case for a given volume, vset related congestion function was getting called which caused node to panic.RESOLUTION:Congestion functions are deprecated in newer linux kernel versions and they are required for MD/DM devices NOT for vxvm.So explicit callback functions are removed and congestion control now refers linux linux standard mechanism.* 4111009 (Tracking ID: 4108475)SYMPTOM:vxfentsthdw script failed with "Expect no writes for disks.."DESCRIPTION:In dmp_return_io() function DMP_SET_BP_ERROR() macro sets DKE_EACCES error on errbp but it is not reflected in errbp->orig_bp.Because orig_bp is not a VxIO buffer because IO is not coming from VxIO here. DMP_BIODONE() is a macro that checks whether the IO buffer (errbp->orig_bp) is a VxIO buffer, if not then it return success even if the IO error occurred here.RESOLUTION:Need to handle this condition to fix this issue, added 2more iodone functions as VxIO signatures to identify the VxIO buffer in vxdmp driver.handled non VxIO buffer case by setting proper error code on the io buffer.* 4111309 (Tracking ID: 4091390)SYMPTOM:vradmind hit the core dump while accessing pHdr, which is already freed.DESCRIPTION:While processing the config message - CFG_UPDATE, we incorrectly freed the existing config message objects. Later, objects are accessed again which dumped the vradmind core.RESOLUTION:Changes are done to access the correct configuration objects.* 4111437 (Tracking ID: 4086131)SYMPTOM:Assert is hit when rlink is upgraded as a part of DG upgrade even though rlink is not encrypted prior to upgrade.DESCRIPTION:during the regression run seeing the vrlink core FMI:====RESOLUTION:Upgrade rlink as a part of dg upgrade only when rlink is encrypted which indirectly confirms the presence of /etc/vx/enc-kms-kmip.conf file.2)Reproduction steps: https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.84.152.49%3A8080%2Fjob%2FVVR_Legacy_Functional_LINUX_Runner%2F1173%2Fconsole&data=05%7C01%7CRavi.Teja%40veritas.com%7C02c506a665f24e09203c08da7e9351e6%7Cfc8e13c0422c4c55b3eaca318e6cac32%7C0%7C0%7C637961468450647119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FA2wzePDqfwzaX1oac5iBF%2FiQQvkzRB%2BAmRhDW9AyEQ%3D&reserved=03) Build details:[root@sclvmesxd06vm15 ~]# rpm -qi VRTSvxvm Name : VRTSvxvmVersion : 8.0.2.0000Release : RHEL7Architecture: x86_64Install Date: Sat Aug 13 15:57:50 2022Group : Applications/SystemSize : 1217454205License : Veritas ProprietarySignature : (none)Source RPM : VRTSvxvm-8.0.2.0000-RHEL7.src.rpmBuild Date : Tue Aug 9 02:47:36 2022Build Host : vxvmrsvrhel7bld1.rsv.ven.veritas.comRelocations : (not relocatable)Packager : enterprise_technical_support@veritas.comVendor : Veritas Technologies LLCURL : www.veritas.com/supportSummary : Veritas Volume Manager* 4111442 (Tracking ID: 4066785)SYMPTOM:When the replicated disks are in SPLIT mode, importing its disk group failed with "Device is a hardware mirror".DESCRIPTION:When the replicated disks are in SPLIT mode, which are readable and writable, importing its disk group failed with "Device is a hardware mirror". Third party doesn't expose disk attribute to show when it's in SPLIT mode. With this new enhancement, the replicated disk group can be imported with option `-o usereplicatedev=only`.RESOLUTION:The code is enhanced to import the replicated disk group with option `-o usereplicatedev=only`.* 4111560 (Tracking ID: 4098391)SYMPTOM:Kernel panic is observed with following stack:#6 [ffffa479c21cf6f0] page_fault at ffffffffb240130e [exception RIP: bfq_bio_bfqg+37] RIP: ffffffffb1e78135 RSP: ffffa479c21cf7a0 RFLAGS: 00010002 RAX: 000000000000001f RBX: 0000000000000000 RCX: ffffa479c21cf860 RDX: ffff8bd779775000 RSI: ffff8bd795b2fa00 RDI: ffff8bd795b2fa00 RBP: ffff8bd78f136000 R8: 0000000000000000 R9: ffff8bd793a5b800 R10: ffffa479c21cf828 R11: 0000000000001000 R12: ffff8bd7796b6e60 R13: ffff8bd78f136000 R14: ffff8bd795b2fa00 R15: ffff8bd7946ad0bc ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018#7 [ffffa479c21cf7b0] bfq_bic_update_cgroup at ffffffffb1e78458#8 [ffffa479c21cf7e8] bfq_bio_merge at ffffffffb1e6f47f#9 [ffffa479c21cf840] blk_mq_submit_bio at ffffffffb1e48c09#10 [ffffa479c21cf8c8] submit_bio_noacct at ffffffffb1e3c7e3#11 [ffffa479c21cf958] submit_bio at ffffffffb1e3c87b#12 [ffffa479c21cf9a8] submit_bh_wbc at ffffffffb1d2536a#13 [ffffa479c21cf9e0] block_read_full_page at ffffffffb1d27ac1#14 [ffffa479c21cfa90] do_read_cache_page at ffffffffb1c2f7e5#15 [ffffa479c21cfb48] read_part_sector at ffffffffb1e546b5#16 [ffffa479c21cfb60] read_lba at ffffffffb1e595d2#17 [ffffa479c21cfba8] efi_partition at ffffffffb1e59f4d#18 [ffffa479c21cfcb8] blk_add_partitions at ffffffffb1e54377#19 [ffffa479c21cfcf8] bdev_disk_changed at ffffffffb1d2a8fa#20 [ffffa479c21cfd30] __blkdev_get at ffffffffb1d2c16c#21 [ffffa479c21cfda0] blkdev_get at ffffffffb1d2c2b4#22 [ffffa479c21cfdb8] __device_add_disk at ffffffffb1e5107e#23 [ffffa479c21cfe20] dmp_register_disk at ffffffffc0e68ae7 [vxdmp]#24 [ffffa479c21cfe50] dmp_reconfigure_db at ffffffffc0e8d8bd [vxdmp]#25 [ffffa479c21cfe80] dmpioctl at ffffffffc0e75cd5 [vxdmp]#26 [ffffa479c21cfe90] dmp_ioctl at ffffffffc0e9d469 [vxdmp]#27 [ffffa479c21cfea8] blkdev_ioctl at ffffffffb1e4ed19#28 [ffffa479c21cfef0] block_ioctl at ffffffffb1d2a719#29 [ffffa479c21cfef8] ksys_ioctl at ffffffffb1cfb262#30 [ffffa479c21cff30] __x64_sys_ioctl at ffffffffb1cfb296#31 [ffffa479c21cff38] do_syscall_64 at ffffffffb1a0538b#32 [ffffa479c21cff50] entry_SYSCALL_64_after_hwframe at ffffffffb240008cDESCRIPTION:VxVM causes kernel panic because of null pointer dereference in kernel code when BFQ disk io scheduler is used. This is observed on SLES15 SP3 minor kernel &gt;= 5.3.18-150300.59.68.1 and SLES15 SP4 minor kernel &gt;= 5.14.21-150400.24.11.1RESOLUTION:Code changes have been done to fix this issue in IS-8.0 and IS-8.0.2.* 4112219 (Tracking ID: 4069134)SYMPTOM:"vxassist maxsize alloc:array:<enclosure_name>" command may fail with below error:VxVM vxassist ERROR V-5-1-18606 No disks match specification for Class: array, Instance: <enclosure_name>DESCRIPTION:If the enclosure name is greater than 16 chars then "vxassist maxsize alloc:array" command can fail. This is because if the enclosure name is more than 16 chars then it gets truncated while copying from VxDMP to VxVM.This further cause the above vxassist command to fail.RESOLUTION:Code changes are done to avoid the truncation of enclosure name while copying from VxDMP to VxVM.* 4112549 (Tracking ID: 4112701)SYMPTOM:Observed reconfig hang on 8 nodes ISO vm setup after rebooting all nodes with a delay of 5min in between them due to Vxconfigd core dumped on master nodeDESCRIPTION:1. reconfig hang on 8 nodes ISO vm setup after rebooting all nodes with a delay of 5min. 2. This is because fork failed during command shipping which caused vxconfigd core dump on master. So all reconfigurations after that failed to process.RESOLUTION:Reboot master node where vold is core dumped.* 4112606 (Tracking ID: 4100069)SYMPTOM:One of standard disk groups fails to auto-import when those disk groups co-exist with the cloned disk group. It failed with below error in syslog.vxvm:vxconfigd[xxx]: V-5-1-569 Disk group <disk group name>, Disk <dmpnode name> Cannot auto-import group:vxvm:vxconfigd[xxx]: #011Disk for disk group not foundDESCRIPTION:The importflags wasn't reset before starting the next disk group import, further caused the next disk group import inherited all the flags from the last round of disk group import. The improper importflags caused the failure.RESOLUTION:Code changes have been made to reset importflags in every round of disk group import.* 4112610 (Tracking ID: 4102532)SYMPTOM:/etc/default/vxsf file gets world write permission when "vxtune storage_connectivity asymmetric" is run.DESCRIPTION:umask for daemon process vxconfigd is 0 and not 0022. This is required for functionality to run properly. For this reason, any file created by vxconfigd gets world-write permission. When "vxtune storage_connectivity asymmetric" is run, a temporary file is created and then it is renamed to vxsf. So vxsf gets world write permission.RESOLUTION:Code changes done so that, instead of default permissions, specific permissions are set to file when file is created. So vxsf does not get world write permission.* 4112884 (Tracking ID: 4107401)SYMPTOM:SRL goes into passthru mode which causes system to run without replicationDESCRIPTION:Issue is seen in FSS environment when new logowner selected after any reconfig is not contributing any storage. If SRL and data volume recovery use different plexes inconsistency is seen while reading SRL data.RESOLUTION:When SRL is recovered do read-write back so that all plexes are consistent.* 4113223 (Tracking ID: 4093067)SYMPTOM:System panicked in the following stack:#9 [] page_fault at [exception RIP: bdevname+26]#10 [] get_dip_from_device [vxdmp]#11 [] dmp_node_to_dip at [vxdmp]#12 [] dmp_check_nonscsi at [vxdmp]#13 [] dmp_probe_required at [vxdmp]#14 [] dmp_check_disabled_policy at [vxdmp]#15 [] dmp_initiate_restore at [vxdmp]#16 [] dmp_daemons_loop at [vxdmp]DESCRIPTION:After got block_device from OS, DMP didn't do the NULL pointer check against block_device->bd_part. This NULL pointer further caused system panic when bdevname() was called.RESOLUTION:The code changes have been done to fix the problem.* 4113225 (Tracking ID: 4068090)SYMPTOM:System panicked in the following stack:#7 page_fault at ffffffffbce010fe[exception RIP: vx_bio_associate_blkg+56]#8 vx_dio_physio at ffffffffc0f913a3 [vxfs]#9 vx_dio_rdwri at ffffffffc0e21a0a [vxfs]#10 vx_dio_read at ffffffffc0f6acf6 [vxfs]#11 vx_read_common_noinline at ffffffffc0f6c07e [vxfs]#12 vx_read1 at ffffffffc0f6c96b [vxfs]#13 vx_vop_read at ffffffffc0f4cce2 [vxfs]#14 vx_naio_do_work at ffffffffc0f240bb [vxfs]#15 vx_naio_worker at ffffffffc0f249c3 [vxfs]DESCRIPTION:To get VxVM volume's block_device from gendisk, VxVM calls bdget_disk(), which increases device inode ref count. The ref count gets decreased by bdput() call that is missed in our code, hence the inode count leaks occurs, which may cause panic in vxfs when issuing IO on VxVM volume.RESOLUTION:The code changes have been done to fix the problem.* 4113310 (Tracking ID: 4114601)SYMPTOM:System gets panicked and rebootedDESCRIPTION:RCA:Start the IO on volume device and pull out it's disk from the machine and hit below panic on RHEL8. dmp_process_errbp dmp_process_errbuf.cold.2+0x328/0x429 [vxdmp] dmpioctl+0x35/0x60 [vxdmp] dmp_flush_errbuf+0x97/0xc0 [vxio] voldmp_errbuf_sio_start+0x4a/0xc0 [vxio] voliod_iohandle+0x43/0x390 [vxio] voliod_loop+0xc2/0x330 [vxio] ? voliod_iohandle+0x390/0x390 [vxio] kthread+0x10a/0x120 ? set_kthread_struct+0x50/0x50As disk pulled out from the machine VxIO hit a IO error and it routed that IO to dmp layer via kernel-kernel IOCTL for error analysis.following is the code path for IO routing,voldmp_errbuf_sio_start()-->dmp_flush_errbuf()--->dmpioctl()--->dmp_process_errbuf()dmp_process_errbuf() retrieves device number of the underlying path (os-device).and it tries to get bdev (i.e. block_device) pointer from path-device number.As path/os-device is removed by disk pull, linux returns fake bdev for the path-device number.For this fake bdev there is no gendisk associated with it (bdev->bd_disk is NULL).We are setting this NULL bdev->bd_disk to the IO buffer routed from vxio.which leads a panic on dmp_process_errbp.RESOLUTION:If bdev->bd_disk found NULL then set DMP_CONN_FAILURE error on the IO buffer and return DKE_ENXIO to vxio driver* 4113328 (Tracking ID: 4102439)SYMPTOM:Customer observed failure When trying to run the vxencrypt rekey operation on an encrypted volume (to perform key rotation).DESCRIPTION:KMS token is of size 64 bytes, we are restricting the token size to 63 bytes and throw an error if the token size is more than 63.RESOLUTION:The issue is resolved by setting the assumption of token size to be size of KMS token, which is 64 bytes.* 4113331 (Tracking ID: 4105565)SYMPTOM:In Cluster Volume Replication(CVR) environment, system panic with below stack when Veritas Volume Replicator(VVR) was doing recovery:[] do_page_fault [] page_fault [exception RIP: volvvr_rvgrecover_nextstage+747] [] volvvr_rvgrecover_done [vxio][] voliod_iohandle[vxio][] voliod_loop at[vxio]DESCRIPTION:There might be a race condition which caused VVR failed to trigger a DCM flush sio. VVR failed to do sanity check against this sio. Hence triggered system panic.RESOLUTION:Code changes have been made to do a sanity check of the DCM flush sio.* 4113342 (Tracking ID: 4098965)SYMPTOM:Vxconfigd dumping Core when scanning IBM XIV Luns with following stack.#0 0x00007fe93c8aba54 in __memset_sse2 () from /lib64/libc.so.6#1 0x000000000061d4d2 in dmp_getenclr_ioctl ()#2 0x00000000005c54c7 in dmp_getarraylist ()#3 0x00000000005ba4f2 in update_attr_list ()#4 0x00000000005bc35c in da_identify ()#5 0x000000000053a8c9 in find_devices_in_system ()#6 0x000000000053aab5 in mode_set ()#7 0x0000000000476fb2 in ?? ()#8 0x00000000004788d0 in main ()DESCRIPTION:This could cause 2 issues if there are more than 1 disk arrays connected:1. If the incorrect memory address exceeds the range of valid virtual memory, it will trigger "Segmentation fault" and crash vxconfigd.2. If the incorrect memory address does not exceed the range of valid virtual memory, it will cause memory corruption issue but maybe not trigger vxconfigd crash issue.RESOLUTION:Code changes have been made to correct the problem.* 4113357 (Tracking ID: 4112433)SYMPTOM:Vulnerabilities have been reported in third party components, [openssl, curl and libxml] that are used by VxVM.DESCRIPTION:Third party components [openssl, curl and libxml] in their current versions, used by VxVM have been reported with security vulnerabilities which needsRESOLUTION:[openssl, curl and libxml] have been upgraded to newer versions in which the reported security vulnerabilities have been addressed.* 4113691 (Tracking ID: 4064772)SYMPTOM:After enabling slub debug, system could hang with IO load.DESCRIPTION:When creating VxVM I/O memory, VxVM does not align the cache size. This unaligned length will be treated as an invalid I/O length in SCSI layer, which causes some I/O requests are stuck in an invalid state and results in the I/Os never being able to complete. Thus system hang could be observed, especially after cache slub debug is enabled.RESOLUTION:Code changes have been done to align the cache size.* 4113692 (Tracking ID: 4089626)SYMPTOM:On RHEL8.5, IO hang occurrs when creating XFS on VxDMP devices or writing file on mounted XFS from VxDMP devices.DESCRIPTION:XFS utilizes chained BIO feature to send BIOs to VxDMP. While the chained BIO isn't suported by VxDMP, hence the BIOs may struck in SCSI disk driver.RESOLUTION:Code changes have been made to support chained BIO.* 4113693 (Tracking ID: 4100775)SYMPTOM:vxconfigd kept waiting for IO drain when removed dmpnodes. It was hung with below stack:[] dmpsync_wait+0xa7/0xf0 [vxdmp][] dmp_destroy_mp_node+0x98/0x120 [vxdmp][] dmp_decode_destroy_dmpnode+0xd3/0x100 [vxdmp][] dmp_decipher_instructions+0x2d7/0x390 [vxdmp][] dmp_process_instruction_buffer+0x1be/0x1e0 [vxdmp][] dmp_reconfigure_db+0x5b/0xe0 [vxdmp][] gendmpioctl+0x76c/0x950 [vxdmp][] dmpioctl+0x39/0x80 [vxdmp][] dmp_ioctl+0x3a/0x70 [vxdmp][] blkdev_ioctl+0x28a/0xa20[] block_ioctl+0x41/0x50[] do_vfs_ioctl+0x3a0/0x5b0[] SyS_ioctl+0xa1/0xc0DESCRIPTION:XFS utilizes chained BIO feature to send BIOs to VxDMP. While the chained BIO isn't supported by VxDMP, it caused VxDMP kept waiting for a completed BIO.RESOLUTION:Code changes have been made to support chained BIO on rhel7.* 4113694 (Tracking ID: 4113323)SYMPTOM:Existing package failed to load on RHEL 8.8 server.DESCRIPTION:RHEL 8.8 is a new release and hence VxVM module is compiled with this new kernel along with few other changes.RESOLUTION:Compiled VxVM code against 8.8 kernel and made changes to make it compatible.* 4114963 (Tracking ID: 4114962)SYMPTOM:File system data corruption with mirrored volumes in Flexible Storage Sharing (FSS) environments during beyond fault storage failure situations.DESCRIPTION:In FSS environments, data change object (DCO) provides functionality to track changes on detached mirrors using bitmaps. This bitmap is later used for re-sync of detached mirrors data (change delta).When DCO volume and data volume share the same set of devices, DCO volumes last mirror failure means IOs on data volume is going to fail. In such cases instead of invaliding DCO volumes, proactively IO is failed.This helps in protecting DCO when entire storage comes back and optimal recovery of mirrors can be performed.When disk for one of the mirror of DCO object become available, the bug in DCO update incorrectly updates metadata of DCO which lead to ignoring valid DCO maps during actual volume recovery and hence newly recovered mirrors of volume missed blocks of valid application data. This lead to corruption when read IO were serviced from the newly recovered mirrors.RESOLUTION:The login of FMR map updating transaction of enabling disks is fixed to resolve the bug. This ensures all valid bitmaps are considered for recovery of mirrors and avoid data loss.* 4115251 (Tracking ID: 4115195)SYMPTOM:Data corruption on file-systems with erasure coded volumesDESCRIPTION:In Erasure coded (EC) volume are used in Flexible shared storage (FSS) environments, data change object (DCO) is used to tracking changes in volume with faulted columns. The DCO provides a bitmap of all changed regions during rebuild of the faulted columns. When EC volume starts with few faulted columns, the log-replay IO could not be performed on those columns. Those additional writes are expected to be tracked in DCO bitmap. Due to bug those IOs are not getting tracked in DCO bitmap as DCO bitmaps are not yet enabled when log-replay is triggered. Hence when the remaining columns are attached back, appropriate data blocks of those log-replay IOs are skipped during rebuild. This leads to data corruption when reads are serviced from those columns.RESOLUTION:Code changes are done to ensure log-replay on EC volume is triggered only after ensuring DCO tracking is enabled. This ensures that all IOs from log-replay operations are tracked in DCO maps for remaining faulted columns of volume.* 4115252 (Tracking ID: 4115193)SYMPTOM:Data corruption on VVR primary with storage loss beyond fault tolerance level in replicated environment.DESCRIPTION:In Flexible Storage Sharing (FSS) environment any node fault can lead to storage failure. In VVR primary when last mirror of SRL (Storage Replicator Log) volume faulted while application writes are in progress replication is expected to go to pass-through mode.This information is persistently recorded in the kernel log (KLOG). In the event of cascaded storage node failures, the KLOG updation protocol could not update quorum number of copies. This mis-match in on-disk v/s in-core state of VVR objects leading to data loss due to missing recovery when all storage faults are resolved.RESOLUTION:Code changes to handle the KLOG update failure in SRL IO failure handling is done to ensure configuration on-disk and in-core is consistent and subsequent application IO will not be allowed to avoid data corruption.* 4115381 (Tracking ID: 4091783)SYMPTOM:Buildarea creation for unixvm were failingDESCRIPTION:unixvm build were failing as there is dependency on the storageapi while creation of build area for unixvm and veki.This intern were causing issues in Veki packaging during unixvm builds and vxio driver compilation dependencyRESOLUTION:Added support for storageapi build area creation and building the storageapi internally from unixvm build scripts.* 4116548 (Tracking ID: 4111254)SYMPTOM:vradmind dumps core with the following stack:#3 0x00007f3e6e0ab3f6 in __assert_fail () from /root/cores/lib64/libc.so.6#4 0x000000000045922c in RDS::getHandle ()#5 0x000000000056ec04 in StatsSession::addHost ()#6 0x000000000045d9ef in RDS::addRVG ()#7 0x000000000046ef3d in RDS::createDummyRVG ()#8 0x000000000044aed7 in PriRunningState::update ()#9 0x00000000004b3410 in RVG::update ()#10 0x000000000045cb94 in RDS::update ()#11 0x000000000042f480 in DBMgr::update ()#12 0x000000000040a755 in main ()DESCRIPTION:vradmind was trying to access a NULL pointer (Remote Host Name) in a rlink object, as the Remote Host attribute of the rlink hasn't been set.RESOLUTION:The issue has been fixed by making code changes.* 4116551 (Tracking ID: 4108913)SYMPTOM:Vradmind dumps core with the following stacks:#3 0x00007f2c171be3f6 in __assert_fail () from /root/coredump/lib64/libc.so.6#4 0x00000000005d7a90 in VList::concat () at VList.C:1017#5 0x000000000059ae86 in OpMsg::List2Msg () at Msg.C:1280#6 0x0000000000441bf6 in OpMsg::VList2Msg () at ../../include/Msg.h:389#7 0x000000000043ec33 in DBMgr::processStatsOpMsg () at DBMgr.C:2764#8 0x00000000004093e9 in process_message () at srvmd.C:418#9 0x000000000040a66d in main () at srvmd.C:733#0 0x00007f4d23470a9f in raise () from /root/core.Jan18/lib64/libc.so.6#1 0x00007f4d23443e05 in abort () from /root/core.Jan18/lib64/libc.so.6#2 0x00007f4d234b3037 in __libc_message () from /root/core.Jan18/lib64/libc.so.6#3 0x00007f4d234ba19c in malloc_printerr () from /root/core.Jan18/lib64/libc.so.6#4 0x00007f4d234bba9c in _int_free () from /root/core.Jan18/lib64/libc.so.6#5 0x00000000005d5a0a in ValueElem::_delete_val () at Value.C:491#6 0x00000000005d5990 in ValueElem::~ValueElem () at Value.C:480#7 0x00000000005d7244 in VElem::~VElem () at VList.C:480#8 0x00000000005d8ad9 in VList::~VList () at VList.C:1167#9 0x000000000040a71a in main () at srvmd.C:743#0 0x000000000040b826 in DList::head () at ../include/DList.h:82#1 0x00000000005884c1 in IpmHandle::send () at Ipm.C:1318#2 0x000000000056e101 in StatsSession::sendUCastStatsMsgToPrimary () at StatsSession.C:1157#3 0x000000000056dea1 in StatsSession::sendStats () at StatsSession.C:1117#4 0x000000000046f610 in RDS::collectStats () at RDS.C:6011#5 0x000000000043f2ef in DBMgr::collectStats () at DBMgr.C:2799#6 0x00007f98ed9131cf in start_thread () from /root/core.Jan26/lib64/libpthread.so.0#7 0x00007f98eca4cdd3 in clone () from /root/core.Jan26/lib64/libc.so.6DESCRIPTION:There is a race condition in vradmind that may cause memory corruption and unpredictable result. Vradmind periodically forks a child thread to collect VVR statistic data and send them to the remote site. While the main thread may also be sending data using the same handler object, thus member variables in the handler object are accessed in parallel from multiple threads and may become corrupted.RESOLUTION:The code changes have been made to fix the issue.* 4116557 (Tracking ID: 4085404)SYMPTOM:Huge perf drop after Veritas Volume Replicator (VVR) entered Data Change Map (DCM) mode, when a large size of Storage Replicator Log (SRL) is configured.DESCRIPTION:The active map flush caused RVG serialization. Once RVG gets serialized, all IOs are queued in restart queue, till the active map flush is finished. The too frequent active map flush caused the huge IO drop during flushing SRL to DCM.RESOLUTION:The code is modified to adjust the frequency of active map flush and balance the application IO and SRL flush.* 4116559 (Tracking ID: 4091076)SYMPTOM:SRL gets into pass-thru mode when it's about to overflow.DESCRIPTION:Primary initiated log search for the requested update sent from secondary. The search aborted with head error as a check condition isn't set correctly.RESOLUTION:Fixed the check condition to resolve the issue.* 4116562 (Tracking ID: 4114257)SYMPTOM:VxVM cmd is hung and file system was waiting for io to complete.file system stack:#3 [] wait_for_completion at #4 [] vx_bc_biowait at [vxfs]#5 [] vx_biowait at [vxfs]#6 [] vx_isumupd at [vxfs]#7 [] __switch_to_asm at #8 [] vx_process_revokedele at [vxfs]#9 [] vx_recv_revokedele at [vxfs]#10 [] vx_recvdele at [vxfs]#11 [] vx_msg_process_thread at [vxfs]vxconfigd stack:[<0>] volsync_wait+0x106/0x180 [vxio][<0>] vol_ktrans+0x9f/0x2c0 [vxio][<0>] volconfig_ioctl+0x82a/0xdf0 [vxio][<0>] volsioctl_real+0x38a/0x450 [vxio][<0>] vols_ioctl+0x6d/0xa0 [vxspec][<0>] vols_unlocked_ioctl+0x1d/0x20 [vxspec]One of vxio thread was waiting for IO drain with below stack. #2 [] schedule_timeout at #3 [] vol_rv_change_sio_start at [vxio] #4 [] voliod_iohandle at [vxio]DESCRIPTION:VVR rvdcm flush SIO was triggered by VVR logowner change and it would set the ru_state throttle flags which caused MDATA_SHIP SIO got queued in rv_mdship_throttleq. As the MDATA_SHIP SIOs are active, it caused rvdcm flush SIO unable to proceed. In the end, rvdcm_flush SIO was waiting for SIOs in rv_mdship_throttleq to complete. SIOs in rv_mdship_throttleq were waiting rvdcm_flush SIO to complete. Hence a dead lock situation.RESOLUTION:Code changes have been made to solve the dead lock issue.* 4116565 (Tracking ID: 4034741)SYMPTOM:Due to a common RVIOmem pool being used by multiple RVG, a deadlock scenario gets created, causing high load average and system hang.DESCRIPTION:The current fix limits IO load on secondary by retaining the updates in NMCOM pool until the DV write done, by which RVIOMEM pool became easy to fill up and deadlock situtaion may occur, esp. when high work load on multiple RVGs or cross direction RVGs.Currently all RVGs share the same RVIOMEM pool, while NMCOM pool, RDBACK pool and network/dv update list are all per-RVGs, so the RVIOMEM pool becomes the bottle neck on secondary, which is easy to full and run into deadlock situation.RESOLUTION:Code changes to honor per-RVG RVIOMEM pool to resolve the deadlock issue.* 4116567 (Tracking ID: 4072862)SYMPTOM:In case RVGLogowner resources get onlined on slave nodes, stop the whole cluster may fail and RVGLogowner resources goes in to offline_propagate state.DESCRIPTION:While stopping whole cluster, the racing may happen between CVM reconfiguration and RVGLogowner change SIO.RESOLUTION:Code changes have been made to fix these racings.* 4116688 (Tracking ID: 4085145)SYMPTOM:System with NVME devices can crash due to memory corruption.DESCRIPTION:As part of changes done to detect NVME devices through IOCTL, extra buflen was sent to nvme ioctl through the VRTSaslapm component.This lead to memory corruption and in some cases can cause system to crash.RESOLUTION:Appropriate code changes have been done in the VRTSaslapm to resolve the memory corruption.* 4117110 (Tracking ID: 4113841)SYMPTOM:VVR panic happened in below code path:kmsg_sys_poll()nmcom_get_next_mblk() nmcom_get_hdr_msg() nmcom_get_next_msg() nmcom_wait_msg_tcp() nmcom_server_main_tcp()DESCRIPTION:When the network scan tool send request to VVR which is unexpected, during the VVR connection handshake, the tcp connection may be terminated immediately by the network scan tool, which may lead to the sock released. Hence, VVR panic when try to refer to it as hit the NULL pointer during the processing.RESOLUTION:The code change has been made to check sock is valid, otherwise, return without continue with the VVR connection.* 4117385 (Tracking ID: 4117350)SYMPTOM:Below error is observed when trying to import # vxdg -n SVOL_SIdg -o useclonedev=on -o updateid import SIdgVxVM vxdg ERROR V-5-1-0 Disk group SIdg: import failed:Replicated dg record is found.Did you want to import hardware replicated LUNs?Try vxdg [-o usereplicatedev=only] import option with -c[s]Please refer to system log for details.DESCRIPTION:REPLICATED flag is used to identify a hardware replicated device so to import dg on the REPLICATED disks , usereplicatedev option must be used . As that was not provided hence issue was observed .RESOLUTION:REPLICATED flag has been removed for Hitachi ShadowImage (SI) disks.* 4118108 (Tracking ID: 4114867)SYMPTOM:Getting these error messages while adding new disks[root@server101 ~]# cat /etc/udev/rules.d/41-VxVM-selinux.rules | tail -1KERNEL=="VxVM*", SUBSYSTEM=="block", ACTION=="add", RUN+="/bin/sh -c 'if [ `/usr/sbin/getenforce` != "Disabled" -a `/usr/sbin/[root@server101 ~]#[root@server101 ~]# systemctl restart systemd-udevd.service[root@server101 ~]# udevadm test /block/sdb 2>&1 | grep "invalid"invalid key/value pair in file /etc/udev/rules.d/41-VxVM-selinux.rules on line 20, starting at character 104 ('D')DESCRIPTION:In /etc/udev/rules.d/41-VxVM-selinux.rules double quotation on Disabled and disable is the issue.RESOLUTION:Code changes have been made to correct the problem.* 4118111 (Tracking ID: 4065490)SYMPTOM:systemd-udev threads consumes more CPU during system bootup or device discovery.DESCRIPTION:During disk discovery when new storage devices are discovered, VxVM udev rules are invoked for creating hardware pathsymbolic link and setting SELinux security context on Veritas device files. For creating hardware path symbolic link to eachstorage device, "find" command is used internally which is CPU intensive operation. If too many storage devices are attached tosystem, then usage of "find" command causes high CPU consumption.Also, for setting appropriate SELinux security context on VxVM device files, restorecon is done irrespective of SELinux is enabled or disabled.RESOLUTION:Usage of "find" command is replaced with "udevadm" command. SELinux security context on VxVM device files is being setonly when SELinux is enabled on system.* 4118845 (Tracking ID: 4116024)SYMPTOM:kernel panicked at gab_ifreemsg with following stack:gab_ifreemsggab_freemsgkmsg_gab_sendvol_kmsg_sendmsgvol_kmsg_senderDESCRIPTION:In a CVR environment there is a RVG of > 600 data volumes, enabling vxvvrstatd daemon through service vxvm-recover. vxvvrstatd calls into ioctl(VOL_RV_APPSTATS) , the latter will generate a kmsg whose length is longer than 64k and trigger a kernel panic due to GAB/LLT no support any message longer than 64k.RESOLUTION:Code changes have been done to add a limitation to the maximum number of data volumes for which that ioctl(VOL_RV_APPSTATS) can request the VVR statistics.* 4119257 (Tracking ID: 4090772)SYMPTOM:vxconfigd/vx commands hang on secondary site in a CVR environment.DESCRIPTION:Due to a window with unmatched SRL positions, if any application (e.g. fdisk) tryingto open the secondary RVG volume will acquire a lock and wait for SRL positions to match.During this if any vxvm transaction kicked in will also have to wait for same lock.Further logowner node panic'd which triggered logownership change protocol which hungas earlier transaction was stuck. As logowner change protocol could not complete,in absence of valid logowner SRL position could not match and caused deadlock. That leadto vxconfigd and vx command hang.RESOLUTION:Added changes to allow read operation on volume even if SRL positions areunmatched. We are still blocking write IOs and just allowing open() call for read-onlyoperations, and hence there will not be any data consistency or integrity issues.* 4119276 (Tracking ID: 4090943)SYMPTOM:On Primary, RLink is continuously getting connected/disconnected with below message seen in secondary syslog: VxVM VVR vxio V-5-3-0 Disconnecting replica <rlink_name> since log is full on secondary.DESCRIPTION:When RVG logowner node panic, RVG recovery happens in 3 phases.At the end of 2nd phase of recovery in-memory and on-disk SRL positions remains incorrectand during this time if there is logowner change then Rlink won't get connected.RESOLUTION:Handled in-memory and on-disk SRL positions correctly.* 4119438 (Tracking ID: 4117985)SYMPTOM:Memory/data corruption hit for EC volumesDESCRIPTION:This is a porting request original request was already reviewed:http://codereview.engba.veritas.com/r/42056/Memory corruption hitting in EC was fixed by calling kernel_fpu_begin() for kernel version < rhel8.6. But in latest kernel kernel_fpu_begin() symbol is not available, We can not use it. So we have created separate Module with name 'storageapi' which is having implementation of _fpu_begin and _fpu_endVxIO module is dependent on 'storageapi'RESOLUTION:take a fpu lock for FPU related operations* 4119553 (Tracking ID: 4118510)SYMPTOM:Volume manager tunable to control log file permissionsDESCRIPTION:With US President Executive Order 14028 compliance changes, all product log file permissions changed to 600. Introduced tunable "log_file_permissions" to control the log file permissions to 600 (default), 640 or 644. The tunable can be changed at install time or any time with reboot.RESOLUTION:Added the log_file_permissions tunable.* 4119852 (Tracking ID: 4081740)SYMPTOM:vxdg flush command slow due to too many luns needlessly access /proc/partitions.DESCRIPTION:Linux BLOCK_EXT_MAJOR(block major 259) is used as extended devt for block devices. When partition number of one device is more than 15, the partition device gets assigned under major 259 to solve the sd limitations (16 minors per device), by which more partitions are allowed for one sd device. During "vxdg flush", for each lun in the disk group, vxconfigd reads file /proc/partitions line by line through fgets() to find all the partition devices with major number 259, which would cause vxconfigd to respond sluggishly if there are large amount of luns in the disk group.RESOLUTION:Code has been changed to remove the needless access on /proc/partitions for the luns without using extended devt.* 4120194 (Tracking ID: 4120191)SYMPTOM:IO hang occurred after getting into DCM mode and flushing SRL to DCM.DESCRIPTION:After getting into DCM mode, upcoming SIOs are throttled during the SRL flush. The throttle is cleared after the SRL flush is done. But in some cases, the SRL flush can't be driven hence the IO hang.RESOLUTION:The code changes have been made to fix the issue.* 4120350 (Tracking ID: 4120878)SYMPTOM:System doesn't come up on taking a reboot after enabling dmp_native_support. System goes into maintenance mode.DESCRIPTION:"vxio.ko" is dependent on the new "storageapi.ko" module. "storageapi.ko" was missing from VxDMP_initrd file, which is created when dmp_native_support is enabled. So on reboot, without "storageapi.ko" present, "vxio.ko" fails to load.RESOLUTION:Code changes have been made to include "strorageapi.ko" in VxDMP_initrd.* 4124887 (Tracking ID: 4090828)SYMPTOM:Dumped fmrmap data for better debuggability for corruption issuesDESCRIPTION:vxplex att/vxvol recover cli will internally fetch fmrmaps from kernel using existing ioctl before starting attach operation and get data in binary format and dump to file and store it with specific format like volname_taskid_date.RESOLUTION:Changes done now dumps the fmrmap data into a binary file.* 4126293 (Tracking ID: 4114927)SYMPTOM:After enabling dmp_native_support and taking reboot, /boot is not mounted VxDMP node.DESCRIPTION:When dmp_native_support is enabled, vxdmproot script is expected to modify the /etc/fstab entry for /boot so that on next boot up, /boot is mounted on dmp device instead of OS device. Also, this operation modifies SELinux context of file /etc/fstab. This causes the machine to go into maintenance mode because of a read permission denied error for /etc/fstab on boot up.RESOLUTION:Code changes have been done to make sure SELinux context is preserved for /etc/fstab file and /boot is mounted on dmp device when dmp_native_support is enabled.* 4127273 (Tracking ID: 4123600)SYMPTOM:while Kms encryption testing panic is generatedDESCRIPTION:vxencrypt command is seen to generate panic crash> btPID: 5445 TASK: ffff92e8e9b09cc0 CPU: 7 COMMAND: "vxencryptd" #0 [ffff9e92d71077b0] machine_kexec at ffffffff9c86c237 #1 [ffff9e92d7107808] __crash_kexec at ffffffff9c9c3c9a #2 [ffff9e92d71078c8] crash_kexec at ffffffff9c9c4e58 #3 [ffff9e92d71078d0] oops_end at ffffffff9c8291db #4 [ffff9e92d71078f0] do_trap at ffffffff9c82596e #5 [ffff9e92d7107940] do_error_trap at ffffffff9c825a25 #6 [ffff9e92d7107980] exc_invalid_op at ffffffff9d3256be #7 [ffff9e92d71079a0] asm_exc_invalid_op at ffffffff9d400af6 [exception RIP: usercopy_abort+116] RIP: ffffffff9d2e8db1 RSP: ffff9e92d7107a58 RFLAGS: 00010246 RAX: 0000000000000070 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff92ecebb998a0 RDI: ffff92ecebb998a0 RBP: 0000000000100000 R8: 0000000000000000 R9: 00000000ffff7fff R10: ffff9e92d7107900 R11: ffffffff9e5e9608 R12: ffff92e8e46d1900 R13: 0000000000000001 R14: 0000000000000001 R15: ffff92e93a400000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #8 [ffff9e92d7107a70] __check_heap_object at ffffffff9cb7f785 #9 [ffff9e92d7107a98] __check_object_size at ffffffff9cbbd2e2#10 [ffff9e92d7107ac0] vol_kio_copy_inout at ffffffffc1452d0c [vxio]#11 [ffff9e92d7107b88] vol_route_io at ffffffffc156df56 [vxio]#12 [ffff9e92d7107bf0] volroute_ioctl at ffffffffc156eb75 [vxio]#13 [ffff9e92d7107d28] volsioctl_real at ffffffffc14eac30 [vxio]#14 [ffff9e92d7107df8] vols_ioctl at ffffffffc0aed4ed [vxspec]#15 [ffff9e92d7107e10] vols_unlocked_ioctl at ffffffffc0aed53d [vxspec]#16 [ffff9e92d7107e18] __x64_sys_ioctl at ffffffff9cbdf1ba#17 [ffff9e92d7107e48] do_syscall_64 at ffffffff9d32515c#18 [ffff9e92d7107eb8] vm_mmap_pgoff at ffffffff9cafe08c#19 [ffff9e92d7107f30] do_syscall_64 at ffffffff9d325169#20 [ffff9e92d7107f50] entry_SYSCALL_64_after_hwframe at ffffffff9d40009b RIP: 00007fa343a3ec6b RSP: 00007fa341ff9cf8 RFLAGS: 00000206 RAX: ffffffffffffffda RBX: 00007fa32c000bc0 RCX: 00007fa343a3ec6b RDX: 00007fa32c000bc0 RSI: 00000000564f4ca1 RDI: 0000000000000004 RBP: 0000000000000000 R8: 0000000000000003 R9: 0000000000000077 R10: 0000000000000073 R11: 0000000000000206 R12: 00007fa32c000b60 R13: 00000000015ee600 R14: 00007fa32c000c7c R15: 0000000000000000 ORIG_RAX: 0000000000000010 CS: 0033 SS: 002bcrash>RESOLUTION:Flag of kmem_cache_usercpy has been added in code as code fix.* 4127934 (Tracking ID: 4124223)SYMPTOM:Core dump is generated for vxconfigd in TC execution.DESCRIPTION:TC creates a scenario where 0s are written in first block of disk. In such case, Null check is necessary in code before some variable is accessed. This Null check is missing which causes vxconfigd core dump in TC execution.RESOLUTION:Necessary Null checks is added in code to avoid vxconfigd core dump.* 4128248 (Tracking ID: 4105204)SYMPTOM:Node not able to join the cluster after iLO "press and hold" scenario in loopDESCRIPTION:- Node is not able to join cluster because newly elected master and surviving slaves are stuck in previous reconfig- This is one of Quorum loss/DG disable scenario- During VCS cleanup of disabled DG, dg deport is triggered which is stuck.- Since dg is anyways disabled due to quorum loss, cluster reboot is needed to come out of situation.- Following vxreconfd stack will be seen on new master and surviving slavesPID: 8135 TASK: ffff9d3e32b05230 CPU: 5 COMMAND: "vxreconfd" #0 [ffff9d3e33c43748] __schedule at ffffffff8f1858da #1 [ffff9d3e33c437d0] schedule at ffffffff8f185d89 #2 [ffff9d3e33c437e0] volsync_wait at ffffffffc349415f [vxio] #3 [ffff9d3e33c43848] _vol_syncwait at ffffffffc3939d44 [vxio] #4 [ffff9d3e33c43870] vol_rwsleep_rdlock_hipri at ffffffffc360e2ab [vxio] #5 [ffff9d3e33c43898] volopenter_hipri at ffffffffc361ae45 [vxio] #6 [ffff9d3e33c438a8] volcvm_ktrans_openter at ffffffffc33ba1e6 [vxio] #7 [ffff9d3e33c438c8] cvm_send_mlocks at ffffffffc33863f8 [vxio] #8 [ffff9d3e33c43910] volmvcvm_cluster_reconfig_exit at ffffffffc3407d1d [vxio] #9 [ffff9d3e33c43940] volcvm_master at ffffffffc33da1b8 [vxio]#10 [ffff9d3e33c439c0] volcvm_vxreconfd_thread at ffffffffc33df481 [vxio]#11 [ffff9d3e33c43ec8] kthread at ffffffff8eac6691#12 [ffff9d3e33c43f50] ret_from_fork_nospec_begin at ffffffff8f192d24RESOLUTION:cluster reboot is needed to come out of situation* 4136205 (Tracking ID: 4111978)SYMPTOM:Replication failed to start due to vxnetd threads not running on secondary site.DESCRIPTION:Vxnetd was waiting to start "nmcomudpsrv" and "nmcomlistenserver" threads. Due to a race condition of some resource between those two thread, vxnetd was stuck in a dead loop till max retry reached.RESOLUTION:Code changes have been made to add lock protection to avoid the race condition.* 4136892 (Tracking ID: 4136458)SYMPTOM:In CVR environment, if CVM slave node is acting as logowner, the DCM resync issues after snapshot restore may hang showing 0% sync is remaining.DESCRIPTION:The DCM resync completion is not correctly communicated to CVM master resulting into hang.RESOLUTION:The DCM resync operation is enhanced to correctly communicate resync completion to CVM master.* 4140218 (Tracking ID: 4117568)SYMPTOM:Vradmind dumps core with the following stack:#1 std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (this=0x7ffdc380d810, __str=<error reading variable: Cannot access memory at address 0x3736656436303563>)#2 0x000000000040e02b in ClientMgr::closeStatsSession#3 0x000000000040d0d7 in ClientMgr::client_ipm_close#4 0x000000000058328e in IpmHandle::~IpmHandle#5 0x000000000057c509 in IpmHandle::events#6 0x0000000000409f5d in mainDESCRIPTION:After terminating vrstat, the StatSession in vradmind was closed and the corresponding Client object was deleted. When closing the IPM object of vrstat, try to access the removed Client, hence the core dump.RESOLUTION:Core changes have been made to fix the issue.* 4149249 (Tracking ID: 4149248)SYMPTOM:Third-party components (OpenSSL, curl, and libxml) used by VxVM exhibit security vulnerabilities.DESCRIPTION:VxVM utilizes current versions of OpenSSL, curl, and libxml, which have been reported to have security vulnerabilities.RESOLUTION:Upgrades to newer versions of OpenSSL, curl, and libxml have been implemented to address the reported security vulnerabilities.* 4162732 (Tracking ID: 4073653)SYMPTOM:After configuring RVGs in async mode on CVR setup with shared storage it is observed that startrep for RVG is failed and vxconfigd hangs on primary master node.DESCRIPTION:Logowner change is hung because DCM read is not completing.RESOLUTION:: Acquisition of mbuf_lock to be done with try method so that vxiods are not kept busy waiting for locks.* 4162734 (Tracking ID: 4098144)SYMPTOM:vxtask list shows the parent process without any sub-tasks which never progresses for SRL volumeDESCRIPTION:vxtask remains stuck since the parent process doesn't exit. It was seen that all childs are completed, but the parent is not able to exit.(gdb) p active_jobs$1 = 1Active jobs are reduced as in when childs complete. Somehow one count is pending and we don't know which child exited without decrementing count. Instrumentation messages are added to capture the issue.RESOLUTION:Added a code that will create a log file in /etc/vx/log/. This file will be deleted when vxrecover exists successfully. The file will be present when vxtask parent hang issue is seen.* 4162735 (Tracking ID: 4132265)SYMPTOM:Machine with NVMe disks panics with following stack: blk_update_requestblk_mq_end_requestdmp_kernel_nvme_ioctldmp_dev_ioctldmp_send_nvme_passthru_cmd_over_nodedmp_pr_do_nvme_readdmp_pgr_readdmpioctldmp_ioctlblkdev_ioctl__x64_sys_ioctldo_syscall_64DESCRIPTION:Issue was applicable to setups with NVMe devices which do not support SCSI3-PR as an ioctl was called without checking correctly if SCSI3-PR was supported.RESOLUTION:Fixed the check to avoid calling the ioctl on devices which do not support SCSI3-PR.* 4162741 (Tracking ID: 4128351)SYMPTOM:System hung observed when switching log owner.DESCRIPTION:VVR mdship SIOs might be throttled due to reaching max allocation count,etc. These SIOs are holding io count. When log owner change kicked in and quiesced RVG. VVR log owner change SIO is waiting for iocount to drop to zero to proceed further. VVR mdship requests from the log client are returned with EAGAIN as RVG quiesced. The throttled mdship SIOs need to be driven by the upcoming mdship requests, hence the deadlock, which caused system hung.RESOLUTION:Code changes have been made to flush the mdship queue before VVR log owner change SIO waiting for IO drain.* 4162742 (Tracking ID: 4122061)SYMPTOM:Observing hung after resync operation, vxconfigd was waiting for slaves' response.DESCRIPTION:VVR logowner was in a transaction and returned VOLKMSG_EAGAIN to CVM_MSG_GET_METADATA which is expected. Once the client received VOLKMSG_EAGAIN, it would sleep 10 jiffies and retry the kmsg . In a busy cluster, it might happen the retried kmsgs plus the new kmsgs got built up and hit the kmsg flowcontrol before the vvr logowner transaction completed. Once the client refused any kmsgs due to the flowcontrol. The transaction on vvr logowner might get stuck because it required kmsg response from all the slave node.RESOLUTION:Code changes have been made to increase the kmsg flowcontrol and don't let kmsg receiver fall asleep but handle the kmsg in a restart function.* 4162743 (Tracking ID: 4087628)SYMPTOM:When DCM is in replication mode with volumes mounted having large regions for DCM to sync and if slave node reboot is triggered, this might cause CVM to go into faulted state .DESCRIPTION:During Resiliency tests, performed sequence of operations as following. 1. On an AWS FSS-CVR setup, replication is started across the sites for 2 RVGs.2. The low owner service groups for both the RVGs are online on a Slave node. 3. Rebooted another Slave node where logowner is not online. 4. After Slave node come back from reboot, it is unable to join CVM Cluster. 5. Also vx commands are also hung/stuck on the CVM Master and Logowner slave node.RESOLUTION:In RU SIO before requesting vxfs_free_region(), drop IO count and hold it again after. Because the transaction has been locked (vol_ktrans_locked = 1) right before calling vxfs_free_region(), we don't need the iocount to hold rvg from being removed.* 4162745 (Tracking ID: 4145063)SYMPTOM:vxio Module fails to load post VxVM package installation.DESCRIPTION:Following message is seen in dmesg:[root@dl360g10-115-v23 ~]# dmesg | grep symbol[ 2410.561682] vxio: no symbol version for storageapi_associate_blkgRESOLUTION:Because of incorrectly nested IF blocks in the "src/linux/kernel/vxvm/Makefile.target", the code for the RHEL 9 block was not getting executed, because of which certain symbols were not present in the vxio.mod.c file. This in turn caused the above mentioned symbol warning to be seen in dmesg.Fixed the improper nesting of the IF conditions.* 4162747 (Tracking ID: 4152014)SYMPTOM:the excluded dmpnodes are visible after system reboot when SELinux is disabled.DESCRIPTION:During system reboot, disks' hardware soft links failed to be created before DMP exclusion in function, hence DMP failed to recognize the excluded dmpnodes.RESOLUTION:Code changes have been made to reduce the latency in creation of hardware soft links and remove tmpfs /dev/vx on an SELinux Disabled platform.* 4162748 (Tracking ID: 4132799)SYMPTOM:If GLM is not loaded, start CVM fails with the following errors:# vxclustadm -m gab startnodeVxVM vxclustadm INFO V-5-2-9687 vxclustadm: Fencing driver is in disabled mode - VxVM vxclustadm ERROR V-5-1-9743 errno 3DESCRIPTION:The error number but the error message is printed while joining CVM fails.RESOLUTION:The code changes have been made to fix the issue.* 4162749 (Tracking ID: 4134790)SYMPTOM:Hardware Replicated DG was marked with clone flag on SLAVEs after failover operation was done on storage side.DESCRIPTION:udid_mismatch are marked on the H/W replicated devices after switched the source storage with the target storage. Disk with udid_mismatch was treated as a clone device. This caused those replicated disks are treated as the cloned disks as well. With clearclone option, Master would remove this flag in the last stage of dg import, but Slaves couldn't. Hence the clone flag was observed on Slaves only.RESOLUTION:The code changes have been made to do extra H/W Replicated disk check.* 4162750 (Tracking ID: 4077944)SYMPTOM:In VVR environment, when I/O throttling gets activated and deactivated by VVR, it may result in an application I/O hang.DESCRIPTION:In case VVR throttles and unthrottles I/O, the diving of throttled I/O is not done in one of the cases.RESOLUTION:Resolved the issue by making sure the application throttled I/Os get driven in all the cases.* 4162751 (Tracking ID: 4132221)SYMPTOM:Supportability requirement for easier path link to dmpdr utilityDESCRIPTION:The current paths of DMPDR utility are so long and hard to remember for the customers. So it was requested to create a symbolic link to this utility for easier access.RESOLUTION:Code changes are made to create a symlink to this utility for easier access* 4162754 (Tracking ID: 4154121)SYMPTOM:When the replicated disks are in SPLIT mode, importing its disk group on target node failed with "Device is a hardware mirror".DESCRIPTION:When the replicated disks are in SPLIT mode, which are readable and writable, importing its disk group on target node failed with "Device is a hardware mirror". Third party doesn't expose disk attribute to show when it's in SPLIT mode. With this new enhancement, the replicated disk group can be imported when enable use_hw_replicatedev.RESOLUTION:The code is enhanced to import the replicated disk group on target node when enable use_hw_replicatedev.* 4162756 (Tracking ID: 4159403)SYMPTOM:When the replicated disks are in SPLIT mode and use_hw_replicatedev is on, disks are marked as cloned disks after the hardware replicated disk group gets imported.DESCRIPTION:add clearclone option automatically when import the hardware replicated disk group to clear the cloned flag on disks.RESOLUTION:The code is enhanced to import the replicated disk group with clearclone option.* 4162757 (Tracking ID: 4160883)SYMPTOM:clone_flag was set on srdf-r1 disks after reboot.DESCRIPTION:Clean clone got reset in case of AUTOIMPORT, which misled the clone_flag got set on the disk in the end.RESOLUTION:Code change has been made to correct the behavior of setting clone_flag on a disk.* 4163989 (Tracking ID: 4162873)SYMPTOM:disk reclaim is slow.DESCRIPTION:Disk reclaim length should be decided by storage's max reclaim length. But Volume Manager split the reclaim request into smaller segments than the maximum reclaim length, which led to a performance regression.RESOLUTION:Code change has been made to avoid splitting the reclaim request in volume manager level.* 4164137 (Tracking ID: 3972344)SYMPTOM:After reboot of a node on a setup where multiple diskgroups / Volumes within diskgroups are present, sometimes in /var/log/messages an error 'vxrecover ERROR V-5-1-11150 Volume <volume_name> does not exist' is logged.DESCRIPTION:In volume_startable function (volrecover.c), dgsetup is called to set the current default diskgroup. This does not update the current_group variable leading to inappropriate mappings. Volumes are searched in an incorrect diskgroup which is logged in the error message.The vxrecover command works fine if the diskgroup name associated with volume is specified. [vxrecover -g <dg_name> -s]RESOLUTION:Changed the code to use switch_diskgroup() instead of dgsetup. Current_group is updated and the current_dg is set. Thus vxrecover finds the Volume correctly.* 4164248 (Tracking ID: 4162349)SYMPTOM:When using vxstat with -S option the values in two columns(MIN(ms) and MAX(ms)) are not printed.DESCRIPTION:When using vxstat with -S option the values in two columns(MIN(ms) and MAX(ms)) are not printed.vxstat -g <dg_name> -i 5 -S -u m OPERATIONS BYTES AVG TIME(ms) MIN(ms) MAX(ms)TYP NAME READ WRITE READ WRITE READ WRITE READ WRITE READ WRITEgf01sxdb320p Mon Apr 22 14:07:55 2024vol admvol 23977 96830 523.707m 425.3496m 1.43 2.12vol appvol 7056 30556 254.3959m 146.1489m 0.85 2.11RESOLUTION:In our code we were not printing the values for last two values. Code changes have been done to fix this issue.* 4164276 (Tracking ID: 4142772)SYMPTOM:In case SRL overflow frequently happens, SRL reaches 99% filled but the rlink is unable to get into DCM mode.DESCRIPTION:When starting DCM mode, need to check if the error mask NM_ERR_DCM_ACTIVE has been set to prevent duplicated triggers. This flag should have been reset after DCM mode was activated by reconnecting the rlink. As there's a racing condition, the rlink reconnect may be completed before DCM is activated, hence the flag isn't able to be cleared.RESOLUTION:The code changes have been made to fix the issue.* 4164312 (Tracking ID: 4133793)SYMPTOM:DCO experience IO Errors while doing a vxsnap restore on vxvm volumes.DESCRIPTION:Dirty flag was getting set in context of an SIO with flag VOLSIO_AUXFLAG_NO_FWKLOG being set. This led to transaction errors while doing a vxsnap restore command in loop for vxvm volumes causing transaction abort. As a result, VxVM tries to cleanup by removing newly added BMs. Now, VxVM tries to access the deleted BMs. however it is not able to since they were deleted previously. This ultimately leads to DCO IO error.RESOLUTION:Skip first write klogging in the context of an IO with flag VOLSIO_AUXFLAG_NO_FWKLOG being set.* 4164693 (Tracking ID: 4149498)SYMPTOM:While upgrading the VxVM package, a number of warnings are seen regarding .ko files not being found for various modules.DESCRIPTION:These warnings are seen because all the unwanted .ko files have been removed.RESOLUTION:Code changes have been done to not get these warnings.* 4164698 (Tracking ID: 4141806)SYMPTOM:TC hung on primary node.DESCRIPTION:1. Secondary sent rlink pause checkpoint request to primary after loading ted spec actions2. Primary received pause checkpoint message from secondary and it didnt process the request because of tedspec action..3. Later on secondary after some timespan, rlink disconnect event occurred due to ack timeout for above pause checkpoint message.4. Above event sent rlink disconnect request to primary which inturn sets rp_disconnected to true.5. This caused primary to continue processing the pause checkpoint message via below code-- vol_rv_service_checkpoint->vol_rv_start_request_processing6. Depending on rlink phase, vol_rv_start_request_processing returns EAGAIN or EBUSY by setting interrupt- VOLRP_INTFLAG_PROCESS_REQUEST which inturn caused RUTHREAD to process the interrupt and set the VOL_RIFLAG_REQUEST_PENDING on rlink via vol_ru_process_interrupts.7. Later pause checkpoint sio tried to send the acknowledgment for VOLRV_MSG_CHECKPOINT message to secondary by setting msgsio_errno to NM_ERR_BUSY without resetting the VOL_RIFLAG_REQUEST_PENDING flag.8. Later after this write pattern is issued on primary and after some time SRL has reached a state where LOG OVERFLOW protection is triggered. This caused the incoming application IOs to throttle continuously till the SRL drains by some amount..9. After this ruthread which does the job of reading the data and sending updates to secondary is not happened as its continuously deferred and not doing any further work as VOL_RIFLAG_REQUEST_PENDING is set which inturn doesnt drain the SRL.10. This caused incoming IO to continuously throttle which caused hang..RESOLUTION:Reset the VOL_RIFLAG_REQUEST_PENDING flag when we try to send ack to secondary in case of EBUSY scenarios I.e when we are not able to process the request currently* 4164704 (Tracking ID: 4136974)SYMPTOM:With multiple RVGs created and replication IP is an IPv6 address, restarting vxnetd results in vxnetd listen sockets binding to IPv4 and hence replication gets disconnected and remains disconnected.DESCRIPTION:With multiple RVGs created and replication IP is an IPv6 address, restarting vxnetd results in vxnetd listen sockets binding to IPv4 and hence replication gets disconnected and remains disconnected.RESOLUTION:Retry test bind on IPv6 for 3 minutes with 5 second delay between the retries* 4167050 (Tracking ID: 4134069)SYMPTOM:VVR replication was not using VxFS SmartMove feature if filesystem was not mounted on RVG Logowner node.DESCRIPTION:Initial synchronization and DCM replay of VVR required the filesystem to be mounted locally on the logowner node as VVR did not have capability to fetch the required information from a remotely mounted filesystem mount point.RESOLUTION:VVR is updated to fetch the required SmartMove related information from a remotely mounted filesystem mount point.* 4167712 (Tracking ID: 4166086)SYMPTOM:VxVM created invalid device symbolic links under root which isn't expected.#ls -al /...lrwxrwxrwx. 1 root root 16 Jun 20 00:20 sdo -> /dev/vx/.dmp/sdolrwxrwxrwx. 1 root root 16 Jun 20 00:27 sdp -> /dev/vx/.dmp/sdplrwxrwxrwx. 1 root root 16 Jun 20 00:21 sdq -> /dev/vx/.dmp/sdqlrwxrwxrwx. 1 root root 16 Jun 20 00:23 sdr -> /dev/vx/.dmp/sdrDESCRIPTION:Volume manager creates invalid device symbolic links by mistake under root.RESOLUTION:The code has been made to create device symbolic links under the right place.* 4167866 (Tracking ID: 4165971)SYMPTOM:Seeing messages while installing VxVM packages.DESCRIPTION:Getting un-expected message on the console while installing VxVM package on the RHEL9 machines.Below are the un_expected messages generated while installing /root/patch/VRTSvxvm* pkg/var/tmp/rpm-tmp.Kl3ycu: line 657: [: missing `]'RESOLUTION:Issue is fixed with the code change.* 4175213 (Tracking ID: 4153457)SYMPTOM:When using Dell/EMC PowerFlex ScaleIO storage, Veritas File System(VxFS) on Veritas Volume Manager(VxVM) volumes fail to mount after reboot.DESCRIPTION:During system boot it was seen that the ScaleIO devices are detected after VxVM has completed it auto discovery of the storage devices. Hence VxVM doesn't auto detect the ScaleIO devices and fail to auto-import the diskgroup and mount the filesystem.RESOLUTION:Appropriate code changes are done to auto discover the ScaleIO devices.Patch ID: VRTSaslapm 8.0.0.2900* 4177622 (Tracking ID: 4177623)SYMPTOM:seen unknown symbolsDESCRIPTION:RHEL8.10 supportRESOLUTION:Made the changes to support RHEL8.10Patch ID: VRTSvxvm-8.0.0.2800* 4093140 (Tracking ID: 4093067)SYMPTOM:System panicked in the following stack:#9 [] page_fault at [exception RIP: bdevname+26]#10 [] get_dip_from_device [vxdmp]#11 [] dmp_node_to_dip at [vxdmp]#12 [] dmp_check_nonscsi at [vxdmp]#13 [] dmp_probe_required at [vxdmp]#14 [] dmp_check_disabled_policy at [vxdmp]#15 [] dmp_initiate_restore at [vxdmp]#16 [] dmp_daemons_loop at [vxdmp]DESCRIPTION:After got block_device from OS, DMP didn't do the NULL pointer check against block_device->bd_part. This NULL pointer further caused system panic when bdevname() was called.RESOLUTION:The code changes have been done to fix the problem.* 4130643 (Tracking ID: 4130642)SYMPTOM:node failed to rejoin the cluster after switched from master to slave due to the failure of the replicated diskgroup import.The below error message could be found in /var/VRTSvcs/log/CVMCluster_A.log.CVMCluster:cvm_clus:monitor:vxclustadm nodestate return code:[101] with output: [state: out of clusterreason: Replicated dg record is found: retry to add a node failed]DESCRIPTION:The flag which shows the diskgroup was imported with usereplicatedev=only failed to be marked since the last time the diskgroup got imported. The missing flag caused the failure of the replicated diskgroup import, further caused node rejoin failure.RESOLUTION:The code changes have been done to flag the diskgroup after it got imported with usereplicatedev=only.* 4132798 (Tracking ID: 4132799)SYMPTOM:If GLM is not loaded, start CVM fails with the following errors:# vxclustadm -m gab startnodeVxVM vxclustadm INFO V-5-2-9687 vxclustadm: Fencing driver is in disabled mode - VxVM vxclustadm ERROR V-5-1-9743 errno 3DESCRIPTION:The error number but the error message is printed while joining CVM fails.RESOLUTION:The code changes have been made to fix the issue.* 4134024 (Tracking ID: 4134023)SYMPTOM:vxconfigrestore(Diskgroup configuration restoration) for H/W Replicated diskgroup failed with below error:# vxconfigrestore -p LINUXSRDFVxVM vxconfigrestore INFO V-5-2-6198 Diskgroup LINUXSRDF configuration restoration started ......VxVM vxdg ERROR V-5-1-0 Disk group LINUXSRDF: import failed:Replicated dg record is found.Did you want to import hardware replicated LUNs?Try vxdg [-o usereplicatedev=only] import option with -c[s]Please refer to system log for details.... ...VxVM vxconfigrestore ERROR V-5-2-3706 Diskgroup configuration restoration for LINUXSRDF failed.DESCRIPTION:H/W Replicated diskgroup can be imported only with option "-o usereplicatedev=only". vxconfigrestore didn't do H/W Replicated diskgroup check, without giving the proper import option diskgroup import failed.RESOLUTION:The code changes have been made to do H/W Replicated diskgroup check in vxconfigrestore .* 4134791 (Tracking ID: 4134790)SYMPTOM:Hardware Replicated DG was marked with clone flag on SLAVEs after failover operation was done on storage side.DESCRIPTION:udid_mismatch are marked on the H/W replicated devices after switched the source storage with the target storage. Disk with udid_mismatch was treated as a clone device. This caused those replicated disks are treated as the cloned disks as well. With clearclone option, Master would remove this flag in the last stage of dg import, but Slaves couldn't. Hence the clone flag was observed on Slaves only.RESOLUTION:The code changes have been made to do extra H/W Replicated disk check.* 4146456 (Tracking ID: 4128351)SYMPTOM:System hung observed when switching log owner.DESCRIPTION:VVR mdship SIOs might be throttled due to reaching max allocation count,etc. These SIOs are holding io count. When log owner change kicked in and quiesced RVG. VVR log owner change SIO is waiting for iocount to drop to zero to proceed further. VVR mdship requests from the log client are returned with EAGAIN as RVG quiesced. The throttled mdship SIOs need to be driven by the upcoming mdship requests, hence the deadlock, which caused system hung.RESOLUTION:Code changes have been made to flush the mdship queue before VVR log owner change SIO waiting for IO drain.* 4146458 (Tracking ID: 4122061)SYMPTOM:Observing hung after resync operation, vxconfigd was waiting for slaves' response.DESCRIPTION:VVR logowner was in a transaction and returned VOLKMSG_EAGAIN to CVM_MSG_GET_METADATA which is expected. Once the client received VOLKMSG_EAGAIN, it would sleep 10 jiffies and retry the kmsg . In a busy cluster, it might happen the retried kmsgs plus the new kmsgs got built up and hit the kmsg flowcontrol before the vvr logowner transaction completed. Once the client refused any kmsgs due to the flowcontrol. The transaction on vvr logowner might get stuck because it required kmsg response from all the slave node.RESOLUTION:Code changes have been made to increase the kmsg flowcontrol and don't let kmsg receiver fall asleep but handle the kmsg in a restart function.* 4146462 (Tracking ID: 4087628)SYMPTOM:When DCM is in replication mode with volumes mounted having large regions for DCM to sync and if slave node reboot is triggered, this might cause CVM to go into faulted state .DESCRIPTION:During Resiliency tests, performed sequence of operations as following. 1. On an AWS FSS-CVR setup, replication is started across the sites for 2 RVGs.2. The low owner service groups for both the RVGs are online on a Slave node. 3. Rebooted another Slave node where logowner is not online. 4. After Slave node come back from reboot, it is unable to join CVM Cluster. 5. Also vx commands are also hung/stuck on the CVM Master and Logowner slave node.RESOLUTION:In RU SIO before requesting vxfs_free_region(), drop IO count and hold it again after. Because the transaction has been locked (vol_ktrans_locked = 1) right before calling vxfs_free_region(), we don't need the iocount to hold rvg from being removed.* 4146472 (Tracking ID: 4115078)SYMPTOM:vxconfigd hung was observed when reboot all nodes of the primary site.DESCRIPTION:When vvr logowner node wasn't configured on Master. VVR recovery was triggered by node leaving, in case data volume was in recovery, vvr logowner would send ilock request to Master node. Master granted the ilock request and sent a response to vvr logonwer. But due to a bug, ilock requesting node id mismatch was detected by vvr logowner. VVR logowner thought the ilock grant failed, mdship IO went into a permanent hang. vxconfigd was stuck and kept waiting for IO drain.RESOLUTION:Code changes have been made to correct the ilock requesting node id in the ilock request in such case.* 4149249 (Tracking ID: 4149248)SYMPTOM:Third-party components (OpenSSL, curl, and libxml) used by VxVM exhibit security vulnerabilities.DESCRIPTION:VxVM utilizes current versions of OpenSSL, curl, and libxml, which have been reported to have security vulnerabilities.RESOLUTION:Upgrades to newer versions of OpenSSL, curl, and libxml have been implemented to address the reported security vulnerabilities.* 4149423 (Tracking ID: 4145063)SYMPTOM:vxio Module fails to load post VxVM package installation.DESCRIPTION:Following message is seen in dmesg:[root@dl360g10-115-v23 ~]# dmesg | grep symbol[ 2410.561682] vxio: no symbol version for storageapi_associate_blkgRESOLUTION:Because of incorrectly nested IF blocks in the "src/linux/kernel/vxvm/Makefile.target", the code for the RHEL 9 block was not getting executed, because of which certain symbols were not present in the vxio.mod.c file. This in turn caused the above mentioned symbol warning to be seen in dmesg.Fixed the improper nesting of the IF conditions.* 4156836 (Tracking ID: 4152014)SYMPTOM:the excluded dmpnodes are visible after system reboot when SELinux is disabled.DESCRIPTION:During system reboot, disks' hardware soft links failed to be created before DMP exclusion in function, hence DMP failed to recognize the excluded dmpnodes.RESOLUTION:Code changes have been made to reduce the latency in creation of hardware soft links and remove tmpfs /dev/vx on an SELinux Disabled platform.* 4156837 (Tracking ID: 4134790)SYMPTOM:Hardware Replicated DG was marked with clone flag on SLAVEs after failover operation was done on storage side.DESCRIPTION:udid_mismatch are marked on the H/W replicated devices after switched the source storage with the target storage. Disk with udid_mismatch was treated as a clone device. This caused those replicated disks are treated as the cloned disks as well. With clearclone option, Master would remove this flag in the last stage of dg import, but Slaves couldn't. Hence the clone flag was observed on Slaves only.RESOLUTION:The code changes have been made to do extra H/W Replicated disk check.* 4156839 (Tracking ID: 4077944)SYMPTOM:In VVR environment, when I/O throttling gets activated and deactivated by VVR, it may result in an application I/O hang.DESCRIPTION:In case VVR throttles and unthrottles I/O, the diving of throttled I/O is not done in one of the cases.RESOLUTION:Resolved the issue by making sure the application throttled I/Os get driven in all the cases.* 4156841 (Tracking ID: 4142054)SYMPTOM:System panicked in the following stack:[ 9543.195915] Call Trace:[ 9543.195938] dump_stack+0x41/0x60[ 9543.195954] panic+0xe7/0x2ac[ 9543.195974] vol_rv_inactive+0x59/0x790 [vxio][ 9543.196578] vol_rvdcm_flush_done+0x159/0x300 [vxio][ 9543.196955] voliod_iohandle+0x294/0xa40 [vxio][ 9543.197327] ? volted_getpinfo+0x15/0xe0 [vxio][ 9543.197694] voliod_loop+0x4b6/0x950 [vxio][ 9543.198003] ? voliod_kiohandle+0x70/0x70 [vxio][ 9543.198364] kthread+0x10a/0x120[ 9543.198385] ? set_kthread_struct+0x40/0x40[ 9543.198389] ret_from_fork+0x1f/0x40DESCRIPTION:- From the SIO stack, we can see that it is a case of done being called twice. - Looking at vol_rvdcm_flush_start(), we can see that when child sio is created, it is being directly added to the the global SIO queue. - This can cause child SIO to start while vol_rvdcm_flush_start() is still in process of generating other child SIOs. - It means that, say the first child SIO gets done, it can find the children count going to zero and calls done.- The next child SIO, also independently find children count to be zero and call done.RESOLUTION:The code changes have been done to fix the problem.Patch ID: VRTSaslapm 8.0.0.2800* 4158230 (Tracking ID: 4158234)SYMPTOM:Support for ASLAPM on RHEL 8.9DESCRIPTION:RHEL 8.9 is new release and hence APM module should be recompiled with new kernel.RESOLUTION:Compiled APM with RHEL 8.9 kernel.Patch ID: VRTSvxvm-8.0.0.2700* 4154821 (Tracking ID: 4149248)SYMPTOM:Third-party components (OpenSSL, curl, and libxml) used by VxVM exhibit security vulnerabilities.DESCRIPTION:VxVM utilizes current versions of OpenSSL, curl, and libxml, which have been reported to have security vulnerabilities.RESOLUTION:Upgrades to newer versions of OpenSSL, curl, and libxml have been implemented to address the reported security vulnerabilities.Patch ID: VRTSvxvm-8.0.0.2600* 4067237 (Tracking ID: 4058894)SYMPTOM:After package installation and reboot , messages regarding udev rules for ignore_device are observed in /var/log/messages .systemd-udevd[774]: /etc/udev/rules.d/40-VxVM.rules:25 Invalid value for OPTIONS key, ignoring: 'ignore_device'DESCRIPTION:From SLES15 Sp3 onwards , ignore_device is deprecated from udev rules and is not available for use anymore . Hence these messages are observed in system logs .RESOLUTION:Required changes have been done to handle this defect.* 4109554 (Tracking ID: 4105953)SYMPTOM:System panic with below stack in CVR environment. #9 [] page_fault at [exception RIP: vol_ru_check_update_done+183]#10 [] vol_rv_write2_done at [vxio]#11 [] voliod_iohandle at [vxio]#12 [] voliod_loop at [vxio]#13 [] kthread atDESCRIPTION:In CVR environment, when IO is issued in writeack sync mode we ack to application when datavolwrite is done on either log client or logowner depending on where IO is issued on. it could happen that VVR freed the metadata I/O update after SRL write is done incase of writeack sync mode, but later after freeing the update, its accessed again and hence we end up in hitting NULL ptr deference.RESOLUTION:Code changes have been made to avoid the accessing NULL pointer.* 4111442 (Tracking ID: 4066785)SYMPTOM:When the replicated disks are in SPLIT mode, importing its disk group failed with "Device is a hardware mirror".DESCRIPTION:When the replicated disks are in SPLIT mode, which are readable and writable, importing its disk group failed with "Device is a hardware mirror". Third party doesn't expose disk attribute to show when it's in SPLIT mode. With this new enhancement, the replicated disk group can be imported with option `-o usereplicatedev=only`.RESOLUTION:The code is enhanced to import the replicated disk group with option `-o usereplicatedev=only`.* 4112549 (Tracking ID: 4112701)SYMPTOM:Observed reconfig hang on 8 nodes ISO vm setup after rebooting all nodes with a delay of 5min in between them due to Vxconfigd core dumped on master nodeDESCRIPTION:1. reconfig hang on 8 nodes ISO vm setup after rebooting all nodes with a delay of 5min. 2. This is because fork failed during command shipping which caused vxconfigd core dump on master. So all reconfigurations after that failed to process.RESOLUTION:Reboot master node where vold is core dumped.* 4113310 (Tracking ID: 4114601)SYMPTOM:System gets panicked and rebootedDESCRIPTION:RCA:Start the IO on volume device and pull out it's disk from the machine and hit below panic on RHEL8. dmp_process_errbp dmp_process_errbuf.cold.2+0x328/0x429 [vxdmp] dmpioctl+0x35/0x60 [vxdmp] dmp_flush_errbuf+0x97/0xc0 [vxio] voldmp_errbuf_sio_start+0x4a/0xc0 [vxio] voliod_iohandle+0x43/0x390 [vxio] voliod_loop+0xc2/0x330 [vxio] ? voliod_iohandle+0x390/0x390 [vxio] kthread+0x10a/0x120 ? set_kthread_struct+0x50/0x50As disk pulled out from the machine VxIO hit a IO error and it routed that IO to dmp layer via kernel-kernel IOCTL for error analysis.following is the code path for IO routing,voldmp_errbuf_sio_start()-->dmp_flush_errbuf()--->dmpioctl()--->dmp_process_errbuf()dmp_process_errbuf() retrieves device number of the underlying path (os-device).and it tries to get bdev (i.e. block_device) pointer from path-device number.As path/os-device is removed by disk pull, linux returns fake bdev for the path-device number.For this fake bdev there is no gendisk associated with it (bdev->bd_disk is NULL).We are setting this NULL bdev->bd_disk to the IO buffer routed from vxio.which leads a panic on dmp_process_errbp.RESOLUTION:If bdev->bd_disk found NULL then set DMP_CONN_FAILURE error on the IO buffer and return DKE_ENXIO to vxio driver* 4113357 (Tracking ID: 4112433)SYMPTOM:Vulnerabilities have been reported in third party components, [openssl, curl and libxml] that are used by VxVM.DESCRIPTION:Third party components [openssl, curl and libxml] in their current versions, used by VxVM have been reported with security vulnerabilities which needsRESOLUTION:[openssl, curl and libxml] have been upgraded to newer versions in which the reported security vulnerabilities have been addressed.* 4114963 (Tracking ID: 4114962)SYMPTOM:File system data corruption with mirrored volumes in Flexible Storage Sharing (FSS) environments during beyond fault storage failure situations.DESCRIPTION:In FSS environments, data change object (DCO) provides functionality to track changes on detached mirrors using bitmaps. This bitmap is later used for re-sync of detached mirrors data (change delta).When DCO volume and data volume share the same set of devices, DCO volumes last mirror failure means IOs on data volume is going to fail. In such cases instead of invaliding DCO volumes, proactively IO is failed.This helps in protecting DCO when entire storage comes back and optimal recovery of mirrors can be performed.When disk for one of the mirror of DCO object become available, the bug in DCO update incorrectly updates metadata of DCO which lead to ignoring valid DCO maps during actual volume recovery and hence newly recovered mirrors of volume missed blocks of valid application data. This lead to corruption when read IO were serviced from the newly recovered mirrors.RESOLUTION:The login of FMR map updating transaction of enabling disks is fixed to resolve the bug. This ensures all valid bitmaps are considered for recovery of mirrors and avoid data loss.* 4115251 (Tracking ID: 4115195)SYMPTOM:Data corruption on file-systems with erasure coded volumesDESCRIPTION:In Erasure coded (EC) volume are used in Flexible shared storage (FSS) environments, data change object (DCO) is used to tracking changes in volume with faulted columns. The DCO provides a bitmap of all changed regions during rebuild of the faulted columns. When EC volume starts with few faulted columns, the log-replay IO could not be performed on those columns. Those additional writes are expected to be tracked in DCO bitmap. Due to bug those IOs are not getting tracked in DCO bitmap as DCO bitmaps are not yet enabled when log-replay is triggered. Hence when the remaining columns are attached back, appropriate data blocks of those log-replay IOs are skipped during rebuild. This leads to data corruption when reads are serviced from those columns.RESOLUTION:Code changes are done to ensure log-replay on EC volume is triggered only after ensuring DCO tracking is enabled. This ensures that all IOs from log-replay operations are tracked in DCO maps for remaining faulted columns of volume.* 4115252 (Tracking ID: 4115193)SYMPTOM:Data corruption on VVR primary with storage loss beyond fault tolerance level in replicated environment.DESCRIPTION:In Flexible Storage Sharing (FSS) environment any node fault can lead to storage failure. In VVR primary when last mirror of SRL (Storage Replicator Log) volume faulted while application writes are in progress replication is expected to go to pass-through mode.This information is persistently recorded in the kernel log (KLOG). In the event of cascaded storage node failures, the KLOG updation protocol could not update quorum number of copies. This mis-match in on-disk v/s in-core state of VVR objects leading to data loss due to missing recovery when all storage faults are resolved.RESOLUTION:Code changes to handle the KLOG update failure in SRL IO failure handling is done to ensure configuration on-disk and in-core is consistent and subsequent application IO will not be allowed to avoid data corruption.* 4115381 (Tracking ID: 4091783)SYMPTOM:Buildarea creation for unixvm were failingDESCRIPTION:unixvm build were failing as there is dependency on the storageapi while creation of build area for unixvm and veki.This intern were causing issues in Veki packaging during unixvm builds and vxio driver compilation dependencyRESOLUTION:Added support for storageapi build area creation and building the storageapi internally from unixvm build scripts.* 4116548 (Tracking ID: 4111254)SYMPTOM:vradmind dumps core with the following stack:#3 0x00007f3e6e0ab3f6 in __assert_fail () from /root/cores/lib64/libc.so.6#4 0x000000000045922c in RDS::getHandle ()#5 0x000000000056ec04 in StatsSession::addHost ()#6 0x000000000045d9ef in RDS::addRVG ()#7 0x000000000046ef3d in RDS::createDummyRVG ()#8 0x000000000044aed7 in PriRunningState::update ()#9 0x00000000004b3410 in RVG::update ()#10 0x000000000045cb94 in RDS::update ()#11 0x000000000042f480 in DBMgr::update ()#12 0x000000000040a755 in main ()DESCRIPTION:vradmind was trying to access a NULL pointer (Remote Host Name) in a rlink object, as the Remote Host attribute of the rlink hasn't been set.RESOLUTION:The issue has been fixed by making code changes.* 4116551 (Tracking ID: 4108913)SYMPTOM:Vradmind dumps core with the following stacks:#3 0x00007f2c171be3f6 in __assert_fail () from /root/coredump/lib64/libc.so.6#4 0x00000000005d7a90 in VList::concat () at VList.C:1017#5 0x000000000059ae86 in OpMsg::List2Msg () at Msg.C:1280#6 0x0000000000441bf6 in OpMsg::VList2Msg () at ../../include/Msg.h:389#7 0x000000000043ec33 in DBMgr::processStatsOpMsg () at DBMgr.C:2764#8 0x00000000004093e9 in process_message () at srvmd.C:418#9 0x000000000040a66d in main () at srvmd.C:733#0 0x00007f4d23470a9f in raise () from /root/core.Jan18/lib64/libc.so.6#1 0x00007f4d23443e05 in abort () from /root/core.Jan18/lib64/libc.so.6#2 0x00007f4d234b3037 in __libc_message () from /root/core.Jan18/lib64/libc.so.6#3 0x00007f4d234ba19c in malloc_printerr () from /root/core.Jan18/lib64/libc.so.6#4 0x00007f4d234bba9c in _int_free () from /root/core.Jan18/lib64/libc.so.6#5 0x00000000005d5a0a in ValueElem::_delete_val () at Value.C:491#6 0x00000000005d5990 in ValueElem::~ValueElem () at Value.C:480#7 0x00000000005d7244 in VElem::~VElem () at VList.C:480#8 0x00000000005d8ad9 in VList::~VList () at VList.C:1167#9 0x000000000040a71a in main () at srvmd.C:743#0 0x000000000040b826 in DList::head () at ../include/DList.h:82#1 0x00000000005884c1 in IpmHandle::send () at Ipm.C:1318#2 0x000000000056e101 in StatsSession::sendUCastStatsMsgToPrimary () at StatsSession.C:1157#3 0x000000000056dea1 in StatsSession::sendStats () at StatsSession.C:1117#4 0x000000000046f610 in RDS::collectStats () at RDS.C:6011#5 0x000000000043f2ef in DBMgr::collectStats () at DBMgr.C:2799#6 0x00007f98ed9131cf in start_thread () from /root/core.Jan26/lib64/libpthread.so.0#7 0x00007f98eca4cdd3 in clone () from /root/core.Jan26/lib64/libc.so.6DESCRIPTION:There is a race condition in vradmind that may cause memory corruption and unpredictable result. Vradmind periodically forks a child thread to collect VVR statistic data and send them to the remote site. While the main thread may also be sending data using the same handler object, thus member variables in the handler object are accessed in parallel from multiple threads and may become corrupted.RESOLUTION:The code changes have been made to fix the issue.* 4116557 (Tracking ID: 4085404)SYMPTOM:Huge perf drop after Veritas Volume Replicator (VVR) entered Data Change Map (DCM) mode, when a large size of Storage Replicator Log (SRL) is configured.DESCRIPTION:The active map flush caused RVG serialization. Once RVG gets serialized, all IOs are queued in restart queue, till the active map flush is finished. The too frequent active map flush caused the huge IO drop during flushing SRL to DCM.RESOLUTION:The code is modified to adjust the frequency of active map flush and balance the application IO and SRL flush.* 4116559 (Tracking ID: 4091076)SYMPTOM:SRL gets into pass-thru mode when it's about to overflow.DESCRIPTION:Primary initiated log search for the requested update sent from secondary. The search aborted with head error as a check condition isn't set correctly.RESOLUTION:Fixed the check condition to resolve the issue.* 4116562 (Tracking ID: 4114257)SYMPTOM:VxVM cmd is hung and file system was waiting for io to complete.file system stack:#3 [] wait_for_completion at #4 [] vx_bc_biowait at [vxfs]#5 [] vx_biowait at [vxfs]#6 [] vx_isumupd at [vxfs]#7 [] __switch_to_asm at #8 [] vx_process_revokedele at [vxfs]#9 [] vx_recv_revokedele at [vxfs]#10 [] vx_recvdele at [vxfs]#11 [] vx_msg_process_thread at [vxfs]vxconfigd stack:[<0>] volsync_wait+0x106/0x180 [vxio][<0>] vol_ktrans+0x9f/0x2c0 [vxio][<0>] volconfig_ioctl+0x82a/0xdf0 [vxio][<0>] volsioctl_real+0x38a/0x450 [vxio][<0>] vols_ioctl+0x6d/0xa0 [vxspec][<0>] vols_unlocked_ioctl+0x1d/0x20 [vxspec]One of vxio thread was waiting for IO drain with below stack. #2 [] schedule_timeout at #3 [] vol_rv_change_sio_start at [vxio] #4 [] voliod_iohandle at [vxio]DESCRIPTION:VVR rvdcm flush SIO was triggered by VVR logowner change and it would set the ru_state throttle flags which caused MDATA_SHIP SIO got queued in rv_mdship_throttleq. As the MDATA_SHIP SIOs are active, it caused rvdcm flush SIO unable to proceed. In the end, rvdcm_flush SIO was waiting for SIOs in rv_mdship_throttleq to complete. SIOs in rv_mdship_throttleq were waiting rvdcm_flush SIO to complete. Hence a dead lock situation.RESOLUTION:Code changes have been made to solve the dead lock issue.* 4116565 (Tracking ID: 4034741)SYMPTOM:Due to a common RVIOmem pool being used by multiple RVG, a deadlock scenario gets created, causing high load average and system hang.DESCRIPTION:The current fix limits IO load on secondary by retaining the updates in NMCOM pool until the DV write done, by which RVIOMEM pool became easy to fill up and deadlock situtaion may occur, esp. when high work load on multiple RVGs or cross direction RVGs.Currently all RVGs share the same RVIOMEM pool, while NMCOM pool, RDBACK pool and network/dv update list are all per-RVGs, so the RVIOMEM pool becomes the bottle neck on secondary, which is easy to full and run into deadlock situation.RESOLUTION:Code changes to honor per-RVG RVIOMEM pool to resolve the deadlock issue.* 4116567 (Tracking ID: 4072862)SYMPTOM:In case RVGLogowner resources get onlined on slave nodes, stop the whole cluster may fail and RVGLogowner resources goes in to offline_propagate state.DESCRIPTION:While stopping whole cluster, the racing may happen between CVM reconfiguration and RVGLogowner change SIO.RESOLUTION:Code changes have been made to fix these racings.* 4117110 (Tracking ID: 4113841)SYMPTOM:VVR panic happened in below code path:kmsg_sys_poll()nmcom_get_next_mblk() nmcom_get_hdr_msg() nmcom_get_next_msg() nmcom_wait_msg_tcp() nmcom_server_main_tcp()DESCRIPTION:When the network scan tool send request to VVR which is unexpected, during the VVR connection handshake, the tcp connection may be terminated immediately by the network scan tool, which may lead to the sock released. Hence, VVR panic when try to refer to it as hit the NULL pointer during the processing.RESOLUTION:The code change has been made to check sock is valid, otherwise, return without continue with the VVR connection.* 4118108 (Tracking ID: 4114867)SYMPTOM:Getting these error messages while adding new disks[root@server101 ~]# cat /etc/udev/rules.d/41-VxVM-selinux.rules | tail -1KERNEL=="VxVM*", SUBSYSTEM=="block", ACTION=="add", RUN+="/bin/sh -c 'if [ `/usr/sbin/getenforce` != "Disabled" -a `/usr/sbin/[root@server101 ~]#[root@server101 ~]# systemctl restart systemd-udevd.service[root@server101 ~]# udevadm test /block/sdb 2>&1 | grep "invalid"invalid key/value pair in file /etc/udev/rules.d/41-VxVM-selinux.rules on line 20, starting at character 104 ('D')DESCRIPTION:In /etc/udev/rules.d/41-VxVM-selinux.rules double quotation on Disabled and disable is the issue.RESOLUTION:Code changes have been made to correct the problem.* 4118111 (Tracking ID: 4065490)SYMPTOM:systemd-udev threads consumes more CPU during system bootup or device discovery.DESCRIPTION:During disk discovery when new storage devices are discovered, VxVM udev rules are invoked for creating hardware pathsymbolic link and setting SELinux security context on Veritas device files. For creating hardware path symbolic link to eachstorage device, "find" command is used internally which is CPU intensive operation. If too many storage devices are attached tosystem, then usage of "find" command causes high CPU consumption.Also, for setting appropriate SELinux security context on VxVM device files, restorecon is done irrespective of SELinux is enabled or disabled.RESOLUTION:Usage of "find" command is replaced with "udevadm" command. SELinux security context on VxVM device files is being setonly when SELinux is enabled on system.* 4118733 (Tracking ID: 4106689)SYMPTOM:Solaris Zones cannot be started due to Method "/lib/svc/method/fs-local" failed with exit status 95. The error logs are observed as below:Mounting ZFS filesystems: cannot mount 'rpool/export' on '/export': directory is not emptycannot mount 'rpool/export' on '/export': directory is not emptycannot mount 'rpool/export/home' on '/export/home': failure mounting parent datasetcannot mount 'rpool/export/home/addm' on /export/home/addm': directory is not empty.... ....svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed: one or more file systems failed.DESCRIPTION:When DMP native support is enabled and the "faulted" zpools are found, VxVM will deport the faulty zpools and re-import them. In case fs-local isn't started before vxvm-startup2, this error handling will cause a non-empty /export which further cause zfs mount failure.RESOLUTION:Code changes have been made to guarantee the mount order of rpool and zpools.* 4118845 (Tracking ID: 4116024)SYMPTOM:kernel panicked at gab_ifreemsg with following stack:gab_ifreemsggab_freemsgkmsg_gab_sendvol_kmsg_sendmsgvol_kmsg_senderDESCRIPTION:In a CVR environment there is a RVG of > 600 data volumes, enabling vxvvrstatd daemon through service vxvm-recover. vxvvrstatd calls into ioctl(VOL_RV_APPSTATS) , the latter will generate a kmsg whose length is longer than 64k and trigger a kernel panic due to GAB/LLT no support any message longer than 64k.RESOLUTION:Code changes have been done to add a limitation to the maximum number of data volumes for which that ioctl(VOL_RV_APPSTATS) can request the VVR statistics.* 4119087 (Tracking ID: 4067191)SYMPTOM:In CVR environment after rebooting Slave node, Master node may panic with below stack:Call Trace:dump_stack+0x66/0x8bpanic+0xfe/0x2d7volrv_free_mu+0xcf/0xd0 [vxio]vol_ru_free_update+0x81/0x1c0 [vxio]volilock_release_internal+0x86/0x440 [vxio]vol_ru_free_updateq+0x35/0x70 [vxio]vol_rv_write2_done+0x191/0x510 [vxio]voliod_iohandle+0xca/0x3d0 [vxio]wake_up_q+0xa0/0xa0voliod_iohandle+0x3d0/0x3d0 [vxio]voliod_loop+0xc3/0x330 [vxio]kthread+0x10d/0x130kthread_park+0xa0/0xa0ret_from_fork+0x22/0x40DESCRIPTION:As part of CVM Master switch a rvg_recovery is triggered. In this step racecondition can occured between the VVR objects due to which the object valueis not updated properly and can cause panic.RESOLUTION:Code changes are done to handle the race condition between VVR objects.* 4119257 (Tracking ID: 4090772)SYMPTOM:vxconfigd/vx commands hang on secondary site in a CVR environment.DESCRIPTION:Due to a window with unmatched SRL positions, if any application (e.g. fdisk) tryingto open the secondary RVG volume will acquire a lock and wait for SRL positions to match.During this if any vxvm transaction kicked in will also have to wait for same lock.Further logowner node panic'd which triggered logownership change protocol which hungas earlier transaction was stuck. As logowner change protocol could not complete,in absence of valid logowner SRL position could not match and caused deadlock. That leadto vxconfigd and vx command hang.RESOLUTION:Added changes to allow read operation on volume even if SRL positions areunmatched. We are still blocking write IOs and just allowing open() call for read-onlyoperations, and hence there will not be any data consistency or integrity issues.* 4119276 (Tracking ID: 4090943)SYMPTOM:On Primary, RLink is continuously getting connected/disconnected with below message seen in secondary syslog: VxVM VVR vxio V-5-3-0 Disconnecting replica <rlink_name> since log is full on secondary.DESCRIPTION:When RVG logowner node panic, RVG recovery happens in 3 phases.At the end of 2nd phase of recovery in-memory and on-disk SRL positions remains incorrectand during this time if there is logowner change then Rlink won't get connected.RESOLUTION:Handled in-memory and on-disk SRL positions correctly.* 4119438 (Tracking ID: 4117985)SYMPTOM:Memory/data corruption hit for EC volumesDESCRIPTION:This is a porting request original request was already reviewed:http://codereview.engba.veritas.com/r/42056/Memory corruption hitting in EC was fixed by calling kernel_fpu_begin() for kernel version < rhel8.6. But in latest kernel kernel_fpu_begin() symbol is not available, We can not use it. So we have created separate Module with name 'storageapi' which is having implementation of _fpu_begin and _fpu_endVxIO module is dependent on 'storageapi'RESOLUTION:take a fpu lock for FPU related operations* 4120350 (Tracking ID: 4120878)SYMPTOM:System doesn't come up on taking a reboot after enabling dmp_native_support. System goes into maintenance mode.DESCRIPTION:"vxio.ko" is dependent on the new "storageapi.ko" module. "storageapi.ko" was missing from VxDMP_initrd file, which is created when dmp_native_support is enabled. So on reboot, without "storageapi.ko" present, "vxio.ko" fails to load.RESOLUTION:Code changes have been made to include "strorageapi.ko" in VxDMP_initrd.* 4121241 (Tracking ID: 4114927)SYMPTOM:After enabling dmp_native_support and taking reboot, /boot is not mounted VxDMP node.DESCRIPTION:When dmp_native_support is enabled, vxdmproot script is expected to modify the /etc/fstab entry for /boot so that on next boot up, /boot is mounted on dmp device instead of OS device. Also, this operation modifies SELinux context of file /etc/fstab. This causes the machine to go into maintenance mode because of a read permission denied error for /etc/fstab on boot up.RESOLUTION:Code changes have been done to make sure SELinux context is preserved for /etc/fstab file and /boot is mounted on dmp device when dmp_native_support is enabled.* 4121714 (Tracking ID: 4081740)SYMPTOM:vxdg flush command slow due to too many luns needlessly access /proc/partitions.DESCRIPTION:Linux BLOCK_EXT_MAJOR(block major 259) is used as extended devt for block devices. When partition number of one device is more than 15, the partition device gets assigned under major 259 to solve the sd limitations (16 minors per device), by which more partitions are allowed for one sd device. During "vxdg flush", for each lun in the disk group, vxconfigd reads file /proc/partitions line by line through fgets() to find all the partition devices with major number 259, which would cause vxconfigd to respond sluggishly if there are large amount of luns in the disk group.RESOLUTION:Code has been changed to remove the needless access on /proc/partitions for the luns without using extended devt.Patch ID: VRTSvxvm-8.0.0.2400* 4110560 (Tracking ID: 4104927)SYMPTOM:vxvm-boot.service fails to start on linux platforms other than SLES15DESCRIPTION:SLES15 specific attribute changes causes vxvm-boot.service to fail to start on other linux platforms.RESOLUTION:A new vxvm-boot.service file to honour vxvm-boot.service for SLES15, the existing vxvm-boot.service file will serve for other linux platforms.* 4113324 (Tracking ID: 4113323)SYMPTOM:Existing package failed to load on RHEL 8.8 server.DESCRIPTION:RHEL 8.8 is a new release and hence VxVM module is compiled with this new kernel along with few other changes.RESOLUTION:Compiled VxVM code against 8.8 kernel and made changes to make it compatible.* 4113661 (Tracking ID: 4091076)SYMPTOM:SRL gets into pass-thru mode when it's about to overflow.DESCRIPTION:Primary initiated log search for the requested update sent from secondary. The search aborted with head error as a check condition isn't set correctly.RESOLUTION:Fixed the check condition to resolve the issue.* 4113663 (Tracking ID: 4095163)SYMPTOM:System panic with below stack: #6 [] invalid_op at [exception RIP: __slab_free+414] #7 [] kfree at #8 [] vol_ru_free_update at [vxio] #9 [] vol_ru_free_updateq at [vxio]#10 [] vol_rv_write2_done at [vxio]#11 [] voliod_iohandle at [vxio]#12 [] voliod_loop at [vxio]DESCRIPTION:The update gets freed as a part of VVR recovery. At the same time, this update also gets freed in VVR second phase of write. Hence there is a race in freeing the updates and caused the system panic.RESOLUTION:Code changes have been made to avoid* 4113664 (Tracking ID: 4091390)SYMPTOM:vradmind hit the core dump while accessing pHdr, which is already freed.DESCRIPTION:While processing the config message - CFG_UPDATE, we incorrectly freed the existing config message objects. Later, objects are accessed again which dumped the vradmind core.RESOLUTION:Changes are done to access the correct configuration objects.* 4113666 (Tracking ID: 4064772)SYMPTOM:After enabling slub debug, system could hang with IO load.DESCRIPTION:When creating VxVM I/O memory, VxVM does not align the cache size. This unaligned length will be treated as an invalid I/O length in SCSI layer, which causes some I/O requests are stuck in an invalid state and results in the I/Os never being able to complete. Thus system hang could be observed, especially after cache slub debug is enabled.RESOLUTION:Code changes have been done to align the cache size.Patch ID: VRTSvxvm-8.0.0.2200* 4058590 (Tracking ID: 4058867)SYMPTOM:Old VxVM rpm fails to load on RHEL8.7 minor kernel 4.18.0-425.10.1.el8_7.x86_64DESCRIPTION:RedHat did some critical changes in latest kernel which causing soft-lockup issue to VxVM kernel modules while installation.RESOLUTION:As suggested by RedHat (https://access.redhat.com/solutions/6985596) VxVM modules compiled with RHEL 8.7 minor kernel.* 4108392 (Tracking ID: 4107802)SYMPTOM:vxdmp fails to load and system hangs.DESCRIPTION:This issue occurs due to changes in the RHEL8.7 minor kernel and incorrect module is calculated for best-fit.RESOLUTION:Modified existing modinst-vxvm script to calculate correct best-fit module.Patch ID: VRTSaslapm-8.0.0.2200* 4108933 (Tracking ID: 4107932)SYMPTOM:Support for ASLAPM on RHEL8.7 minor kernel 4.18.0-425.10.1.el8_7.x86_64DESCRIPTION:RedHat did some critical changes in latest kernel which causing soft-lockup issue kernel modules while installation.RESOLUTION:As suggested by RedHat (https://access.redhat.com/solutions/6985596) modules compiled with RHEL 8.7 minor kernel.Patch ID: VRTSaslapm-8.0.0.2100* 4102502 (Tracking ID: 4102501)SYMPTOM:A security vulnerability exists in the third-party component libcurl.DESCRIPTION:VxVM uses a third-party component named libcurl in which a security vulnerability exists.RESOLUTION:VxVM is updated to use a newer version of libcurl in which the security vulnerability has been addressed.Patch ID: -8.0.0.1900* 4102924 (Tracking ID: 4101128)SYMPTOM:Old VxVM rpm fails to load on RHEL8.7DESCRIPTION:The RHEL8.7 is a new OS release and has multiple kernel changes which were making VxVM incompatible with OS kernel version 4.18.0-425.3.1RESOLUTION:Required code changes have been done. VxVM module compiled with RHEL 8.7 kernel.Patch ID: VRTSaslapm-8.0.0.1900* 4102973 (Tracking ID: 4101139)SYMPTOM:Support for ASLAPM on RHEL 8.7 kernelDESCRIPTION:The RHEL8.7 is new release and hence APM module should be recompiled with new kernel.RESOLUTION:Compiled APM with new kernel.Patch ID: -8.0.0.1800* 4067609 (Tracking ID: 4058464)SYMPTOM:vradmin resizevol fails when FS is not mounted on master.DESCRIPTION:vradmin resizevol cmd resizes datavolume, FS on the primary site whereas on the secondary site it resizes only datavolume as FS is not mounted on the secondary site. vradmin resizevol cmd ships the cmd to logowner at vradmind level and vradmind on logowner in turn tries to ship the lowlevel vxcommands to master at vradmind level and then finally cmd gets executed on master.RESOLUTION:Changes introduced to ship the cmd to the node on which FS is mounted. cvm nodename must be provided where FS gets mounted which is then used by vradmind to ship cmd to that respective mounted node.* 4067635 (Tracking ID: 4059982)SYMPTOM:In container environment, vradmin migrate cmd fails multiple times due to rlink not in connected state.DESCRIPTION:In VVR, rlinks are disconnected and connected back during the process of replication lifecycle. And, in this mean time when vradmin migrate cmd gets executed it experience errors. It internally causes vradmind to make configuration changes multiple times which impact further vradmin commands getting executed.RESOLUTION:vradmin migrate cmd requires rlink data to be up-to-date on both primary and secondary. It internally executes low-level cmds like vxrvg makesecondary and vxrvg makeprimary to change the role of primary and secondary. These cmds doesn't depend on rlink to be in connected state. Changes are done to remove the rlink connection handling.* 4070098 (Tracking ID: 4071345)SYMPTOM:Replication is unresponsive after failed site is up.DESCRIPTION:Autosync and unplanned fallback synchronisation had issues in a mix of cloud and non-cloud Volumes in RVG. After a cloud volume is found rest of the volumes were getting ignored for synchronisationRESOLUTION:Fixed condition to make it iterate over all Volumes.* 4078531 (Tracking ID: 4075860)SYMPTOM:On RHEL8 NBFS/Access commands like python3, sort, sudo, ssh, etc are generating core dump during execution of the command mkfs.vxfs mkfs.ext4 in parallelDESCRIPTION:On RHEL8 NBFS/Access commands like python3, sort, sudo, ssh, etc are generating core dump during execution of the command mkfs.vxfs mkfs.ext4 in parallel. This was happening due to missing fpu armor protection for FPU instruction set.RESOLUTION:Fix is added to use FPU protection while using FPU instruction set* 4079345 (Tracking ID: 4069940)SYMPTOM:FS mount failed during Cluster configuration on 24-node physical BOM setup.DESCRIPTION:FS mount failed during Cluster configuration on 24-node physical BOM setup due to vxvm transactions were taking time more that vcs timeouts.RESOLUTION:Fix is added to reduce unnecessary transaction time on large node setup.* 4080041 (Tracking ID: 4056953)SYMPTOM:3PAR PE LUNs are reported in error state by 3PAR ASLDESCRIPTION:3PAR storage presents some special STORAGE LUNs(3PAR PE) and these need to be SKIPPED by VxVM and not claimed. This causes an issue for VxDMP to handle as multiple PE LUNs from different 3PAR enclosures.RESOLUTION:Fix added to SKIP the 3PAR PE luns by 3PAR ASL to avoid disks being reported in error state.* 4080105 (Tracking ID: 4045837)SYMPTOM:DCL volume subdisks doesnot relocate after node fault timeout and remains in RELOCATE stateDESCRIPTION:If DCO has failed plexs and dco is on different disks than data, dco relocation need to be triggered explicitly as try_fss_reloc will only perform dco relocation in context of data which may not succeed if sufficient data disks not available (additional host/disks may be available where dco can relocate)RESOLUTION:Fix is added to relocate DCL subdisks to available spare disks* 4080122 (Tracking ID: 4044068)SYMPTOM:Replace Node is failing at Configuring NetBackup stage due to vxdisk init failed with error "Could not obtain requested lock".DESCRIPTION:Replace Node is failing at Configuring NetBackup stage due to vxdisk init failed with error "Could not obtain requested lock".RESOLUTION:Fix is added to retry transaction few times if it fails with this error* 4080269 (Tracking ID: 4044898)SYMPTOM:we were unable to see rlink tags from info records with the vxrlink listtag command.DESCRIPTION:Making rlinks FIPS compliant has 2nd phase in which we are dealing with disk group upgrade path, where rlink enc tags needs to be copied to info record and needs to be FIPS compliant one. here vxdg upgrade will internally call vxrlink and vxencrypt to upgrade the rlink and rekey the rlink keys respectively.RESOLUTION:copied all the encryption tags for rlink to info record and when we are upgrading DG we will internally upgrade the rlink, this upgradation process will copy rlink tags to info records.* 4080276 (Tracking ID: 4065145)SYMPTOM:During addsec we were unable to processencrypted volume tags for multiple volumes and vsets. Error we saw: $ vradmin -g dg2 -encrypted addsec dg2_rvg1 10.210.182.74 10.210.182.75 Error: Duplicate tag name vxvm.attr.enckeytype provided in input.DESCRIPTION:The number of tags was not defined and we were processing all the tags at a time instead of processing max number of tags for a volume.RESOLUTION:Introduced a number of tags variable depend on the cipher method (CBC/GCM), as well fixed minor code issues.* 4080277 (Tracking ID: 3966157)SYMPTOM:the feature of SRL batching was broken and we were not able to enable it as it might caused problems.DESCRIPTION:Batching of updates needs to be done as to get benefit of batching multiple updates and getting performance increasedRESOLUTION:we have decided to simplify the working as we are now aligning each of the small update within a total batch to 4K size so that, by default we will get the whole batch aligned one, and then there is no need of book keeping for last update and hence reducing the overhead of different calculations. we are padding individual updates to reduce overhead of book keeping things around last update in a batch, by padding each updates to 4k, we will be having a batch of updates which is 4k aligned itself.* 4080579 (Tracking ID: 4077876)SYMPTOM:When one cluster node is rebooted, EC log replay is triggered for shared EC volume. It is seen that system is crashed during this EC log replay.DESCRIPTION:It is seen that two flags are assigned same value. So, system panicked during flag check.RESOLUTION:Changed the code flow to avoid checking values of flags having same value.* 4080845 (Tracking ID: 4058166)SYMPTOM:While setting up VVR/CVR on large size data volumes (size > 3TB) with filesystems mounted on them, initial autosync operation takes a lot of time to complete.DESCRIPTION:While performing autosync on VVR/CVR setup for a volume with filesystem mounted, if smartmove feature is enabled, the operation does smartsync by syncing only the regions dirtied by filesystem, instead of syncing entire volume, which completes faster than normal case. However, for large size volumes (size > 3TB), smartmove feature does not get enabled, even with filesystem mounted on them and hence autosync operation syncs entire volume. This behaviour is due to smaller size DCM plexes allocated for such large size volumes, autosync ends up performing complete volume sync,taking lot more time to complete.RESOLUTION:Increase the limit of DCM plex size (loglen) beyond 2MB so that smart move feature can be utilised properly.* 4080846 (Tracking ID: 4058437)SYMPTOM:Replication between 8.0 and 7.4.x fails with an error due to sector size field.DESCRIPTION:7.4.x branch has sectorsize set to zero which internally is indicated as 512 byte. It caused the startrep, resumerep to fail with the below error message. Message from Primary: VxVM VVR vxrlink ERROR V-5-1-20387 sector size mismatch, Primary is having sector size 512, Secondary is having sector size 0RESOLUTION:A check added to support replication between 8.0 and 7.4.x* 4081790 (Tracking ID: 4080373)SYMPTOM:SFCFSHA configuration failed on RHEL 8.4 due to 'chmod -R' error.DESCRIPTION:Failure messages are getting logged as all log permissions are changed to 600 during the upgrade and all log files moved to '/var/log/vx'.RESOLUTION:Added -f option to chmod command to suppress warning and redirect errors from mv command to /dev/null.* 4083337 (Tracking ID: 4081890)SYMPTOM:On RHEL8 NBFS/Access commands like python3, sort, sudo, ssh, etc are generating core dump during execution of the command mkfs.vxfs mkfs.ext4 in parallel.DESCRIPTION:On RHEL8 NBFS/Access commands like python3, sort, sudo, ssh, etc are generating core dump during execution of the command mkfs.vxfs mkfs.ext4 in parallel. This was happening due to missing fpu armor protection for FPU instruction set.RESOLUTION:Fix is added to use FPU protection while using FPU instruction set* 4085619 (Tracking ID: 4086718)SYMPTOM:VxVM fails to install because vxdmp module fails to load on latest minor kernel of SLES15SP2.DESCRIPTION:VxVM modules fail to load on latest minor kernel of SLES15SP2. Following messages can be seen logged in system logs: vxvm-boot[32069]: ERROR: No appropriate modules found. vxvm-boot[32069]: Error in loading module "vxdmp". See documentation. vxvm-boot[32069]: Modules not LoadedRESOLUTION:Code changes have been done to fix this issue.* 4087233 (Tracking ID: 4086856)SYMPTOM:For Appliance FLEX product using VRTSdocker-plugin, docker.service needs to be replaced as it is not supported on RHEL8.DESCRIPTION:Appliance FLEX product using VRTSdocker-plugin is switching to RHEL8 on which docker.service does not exist. vxinfoscale-docker.service must stop after all container services are stopped. podman.service shuts down after all container services are stopped, so docker.service can be replaced with podman.service.RESOLUTION:Added platform-specific dependencies for VRTSdocker-plugin. For RHEL8 podman.service introduced.* 4087439 (Tracking ID: 4088934)SYMPTOM:"dd" command on a simple volume results in kernel panic.DESCRIPTION:Kernel panic is observed with following stack trace: #0 [ffffb741c062b978] machine_kexec at ffffffffa806fe01 #1 [ffffb741c062b9d0] __crash_kexec at ffffffffa815959d #2 [ffffb741c062ba98] crash_kexec at ffffffffa815a45d #3 [ffffb741c062bab0] oops_end at ffffffffa8036d3f #4 [ffffb741c062bad0] general_protection at ffffffffa8a012c2 [exception RIP: __blk_rq_map_sg+813] RIP: ffffffffa84419dd RSP: ffffb741c062bb88 RFLAGS: 00010202 RAX: 0c2822c2621b1294 RBX: 0000000000010000 RCX: 0000000000000000 RDX: ffffb741c062bc40 RSI: 0000000000000000 RDI: ffff8998fc947300 RBP: fffff92f0cbe6f80 R8: ffff8998fcbb1200 R9: fffff92f0cbe0000 R10: ffff8999bf4c9818 R11: 000000000011e000 R12: 000000000011e000 R13: fffff92f0cbe0000 R14: 00000000000a0000 R15: 0000000000042000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #5 [ffffb741c062bc38] scsi_init_io at ffffffffc03107a2 [scsi_mod] #6 [ffffb741c062bc78] sd_init_command at ffffffffc056c425 [sd_mod] #7 [ffffb741c062bcd8] scsi_queue_rq at ffffffffc0311f6e [scsi_mod] #8 [ffffb741c062bd20] blk_mq_dispatch_rq_list at ffffffffa8447cfe #9 [ffffb741c062bdc0] __blk_mq_do_dispatch_sched at ffffffffa844cae0 #10 [ffffb741c062be28] __blk_mq_sched_dispatch_requests at ffffffffa844d152 #11 [ffffb741c062be68] blk_mq_sched_dispatch_requests at ffffffffa844d290 #12 [ffffb741c062be78] __blk_mq_run_hw_queue at ffffffffa84466a3 #13 [ffffb741c062be98] process_one_work at ffffffffa80bcd74 #14 [ffffb741c062bed8] worker_thread at ffffffffa80bcf8d #15 [ffffb741c062bf10] kthread at ffffffffa80c30ad #16 [ffffb741c062bf50] ret_from_fork at ffffffffa8a001ffRESOLUTION:Code changes have been done to fix this issue.* 4087791 (Tracking ID: 4087770)SYMPTOM:Data corruption post mirror attach operation seen after complete storage fault for DCO volumes.DESCRIPTION:DCO (data change object) tracks delta changes for faulted mirrors. During complete storage loss of DCO volume mirrors in, DCO object will be marked as BADLOG and becomes unusable for bitmap tracking. Post storage reconnect (such as node rejoin in FSS environments) DCO will be re-paired for subsequent tracking. During this if VxVM finds any of the mirrors detached for data volumes, those are expected to be marked for full-resync as bitmap in DCO has no valid information. Bug in repair DCO operation logic prevented marking mirror for full-resync in cases where repair DCO operation is triggered before data volume is started. This resulted into mirror getting attached without any data being copied from good mirrors and hence reads serviced from such mirrors have stale data, resulting into file-system corruption and data loss.RESOLUTION:Code has been added to ensure repair DCO operation is performed only if volume object is enabled so as to ensure detached mirrors are marked for full-resync appropriately.* 4088076 (Tracking ID: 4054685)SYMPTOM:RVG recovery gets hung in case of reconfiguration scenarios in CVR environments leading to vx commands hung on master node.DESCRIPTION:As a part of rvg recovery we perform DCM, datavolume recovery. But datavolume recovery takes long time due to wrong IOD handling done in linux platforms.RESOLUTION:Fix the IOD handling mechanism to resolve the rvg recovery handling.* 4088483 (Tracking ID: 4088484)SYMPTOM:DMP_APM module is not getting loaded and throwing following message in the dmesg logs: Mod load failed for dmpnvme module: dependency conflict VxVM vxdmp V-5-0-1015 DMP_APM: DEPENDENCY CONFLICTDESCRIPTION:NVMe module loading failed as dmpaa module dependency added in APM and system doesn't have any A/A type disk which inturn nvme module load failed.RESOLUTION:Removed A/A dependency from NVMe APM.* 4088762 (Tracking ID: 4087099)SYMPTOM:DG is not imported after upgrade to InfoScale 8.0u1 on RHEL8.6 and NVME disks are in an error state.DESCRIPTION:NVME disks minor number was getting changed when scandisks was performed. This was leading to incorrect major / minor information being present in vold of the core database.RESOLUTION:Fixed device open by passing O_RDONLY. With write permissions it was changing minor number.Patch ID: VRTSaslapm-8.0.0.1800* 4080041 (Tracking ID: 4056953)SYMPTOM:3PAR PE LUNs are reported in error state by 3PAR ASLDESCRIPTION:3PAR storage presents some special STORAGE LUNs(3PAR PE) and these need to be SKIPPED by VxVM and not claimed. This causes an issue for VxDMP to handle as multiple PE LUNs from different 3PAR enclosures.RESOLUTION:Fix added to SKIP the 3PAR PE luns by 3PAR ASL to avoid disks being reported in error state.* 4088762 (Tracking ID: 4087099)SYMPTOM:DG is not imported after upgrade to InfoScale 8.0u1 on RHEL8.6 and NVME disks are in an error state.DESCRIPTION:NVME disks minor number was getting changed when scandisks was performed. This was leading to incorrect major / minor information being present in vold of the core database.RESOLUTION:Fixed device open by passing O_RDONLY. With write permissions it was changing minor number.* 4081684 (Tracking ID: 4082799)SYMPTOM:A security vulnerability exists in the third-party component libcurl.DESCRIPTION:VxVM uses a third-party component named libcurl in which a security vulnerability exists.RESOLUTION:VxVM is updated to use a newer version of libcurl in which the security vulnerability has been addressed.Patch ID: -8.0.0.1600* 4057420 (Tracking ID: 4060462)SYMPTOM:System is unresponsive while adding new nodes.DESCRIPTION:After a node is removed, and adding node with different node name is attempted; system turns unresponsive. When a node leaves a cluster, in-memory information related to the node is not cleared due to the race condition.RESOLUTION:Fixed race condition to clear in-memory information of the node that leaves the cluster.* 4062799 (Tracking ID: 4064208)SYMPTOM:Node is unresponsive while it gets added to the cluster.DESCRIPTION:While a node joins the cluster, if bits on the node are upgraded; size of the object is interpreted incorrectly. Issue is observed when number of objects is higher and on InfoScale 7.3.1 and above.RESOLUTION:Correct sizes are calculated for the data received from the master node.* 4065841 (Tracking ID: 4065495)SYMPTOM:This is new array and we need to add support for EMC PowerStore.DESCRIPTION:EMC PowerStore is new array and current ASL doesn't support it. So it will not be claimed with current ASL. This array support has been now added in the current ASL.RESOLUTION:Code changes to support EMC PowerStore have been done.* 4066213 (Tracking ID: 4052580)SYMPTOM:Multipathing not supported for NVMe devices under VxVM.DESCRIPTION:NVMe devices being non-SCSI devices, are not considered for multipathing.RESOLUTION:Changes introduced to support multipathing for NVMe devices.* 4068407 (Tracking ID: 4068404)SYMPTOM:We need to add support to claim ALUA Disks on HPE 3PAR/Primera/Alletra 9000 arrays.DESCRIPTION:Current ASL doesn't support HPE 3PAR/Primera/Alletra 9000 ALUA array type. This ALUA array support has been now added in the current ASL.RESOLUTION:Code changes to support HPE 3PAR/Primera/Alletra 9000 ALUA array have been done.Patch ID: VRTSaslapm-8.0.0.1600* 4065841 (Tracking ID: 4065495)SYMPTOM:This is new array and we need to add support for EMC PowerStore.DESCRIPTION:EMC PowerStore is new array and current ASL doesn't support it. So it will not be claimed with current ASL. This array support has been now added in the current ASL.RESOLUTION:Code changes to support EMC PowerStore have been done.* 4068407 (Tracking ID: 4068404)SYMPTOM:We need to add support to claim ALUA Disks on HPE 3PAR/Primera/Alletra 9000 arrays.DESCRIPTION:Current ASL doesn't support HPE 3PAR/Primera/Alletra 9000 ALUA array type. This ALUA array support has been now added in the current ASL.RESOLUTION:Code changes to support HPE 3PAR/Primera/Alletra 9000 ALUA array have been done.Patch ID: -8.0.0.1200* 4066259 (Tracking ID: 4062576)SYMPTOM:When hastop -local is used to stop the cluster, dg deport command hangs. Below stack trace is observed in system logs : #0 [ffffa53683bf7b30] __schedule at ffffffffa834a38d #1 [ffffa53683bf7bc0] schedule at ffffffffa834a868 #2 [ffffa53683bf7bd0] blk_mq_freeze_queue_wait at ffffffffa7e4d4e6 #3 [ffffa53683bf7c18] blk_cleanup_queue at ffffffffa7e433b8 #4 [ffffa53683bf7c30] vxvm_put_gendisk at ffffffffc3450c6b [vxio] #5 [ffffa53683bf7c50] volsys_unset_device at ffffffffc3450e9d [vxio] #6 [ffffa53683bf7c60] vol_rmgroup_devices at ffffffffc3491a6b [vxio] #7 [ffffa53683bf7c98] voldg_delete at ffffffffc34932fc [vxio] #8 [ffffa53683bf7cd8] vol_delete_group at ffffffffc3494d0d [vxio] #9 [ffffa53683bf7d18] volconfig_ioctl at ffffffffc3555b8e [vxio] #10 [ffffa53683bf7d90] volsioctl_real at ffffffffc355fc8a [vxio] #11 [ffffa53683bf7e60] vols_ioctl at ffffffffc124542d [vxspec] #12 [ffffa53683bf7e78] vols_unlocked_ioctl at ffffffffc124547d [vxspec] #13 [ffffa53683bf7e80] do_vfs_ioctl at ffffffffa7d2deb4 #14 [ffffa53683bf7ef8] ksys_ioctl at ffffffffa7d2e4f0 #15 [ffffa53683bf7f30] __x64_sys_ioctl at ffffffffa7d2e536DESCRIPTION:This issue is seen due to some updation from kernel side w.r.t to handling request queue.Existing VxVM code set the request handling area (make_request_fn) as vxvm_gen_strategy, this functionality is getting impacted.RESOLUTION:Code changes are added to handle the request queues using blk_mq_init_allocated_queue.Patch ID: -8.0.0.1100* 4064786 (Tracking ID: 4053230)SYMPTOM:RHEL 8.5 support is to be provided with IS 8.0DESCRIPTION:RHEL 8.5 ZDS support is being provided with IS 8.0RESOLUTION:VxVM packages are available with RHEL 8.5 compatibility* 4065628 (Tracking ID: 4065627)SYMPTOM:VxVM modules are not loaded after OS upgrade followed by a reboot .DESCRIPTION:Once the stack installation is completed with configuration , after OS upgrade vxvm directory is not formed under /lib/modules/&amp;lt;upgraded_kernel&amp;gt;veritas/ .RESOLUTION:VxVM code is updated with the required changes .Patch ID: VRTScavf-8.0.0.3300* 4162960 (Tracking ID: 4153873)SYMPTOM:CVM master reboot resulted in volumes disabled on the slave nodeDESCRIPTION:The Infoscale stack exhibits unpredictable behaviour during reboots, sometimes the node hangs to come online, the working node goes into the faulted state and sometimes the cvm won't start on the rebooted node.RESOLUTION:Now we have added the mechanism for making decisions about deport and the code has been integrated with an offline routine.Patch ID: VRTScavf-8.0.0.2800* 4118779 (Tracking ID: 4074274)SYMPTOM:DR test and failover activity might not succeed for hardware-replicated disk groups.DESCRIPTION:In case of hardware-replicated disks like EMC SRDF, failover of disk groups might not automatically succeed and a manual intervention might be needed. After failover, disks at the new primary site have the 'udid_mismatch' flag which needs to be updated manually for a successful failover.RESOLUTION:For DMP environments, the VxVM & DMP extended attributes need to be refreshed by using 'vxdisk scandisks' prior to import. VxVM has also provided a new vxdg import option '-o usereplicatedev=only' with DMP. This option selects only the hardware-replicated disks during LUN selection process.Sample main.cf configuration for DMP managed hardware Replicated disks.CVMVolDg srdf_dg (CVMDiskGroup = LINUXSRDFCVMVolume = { scott, scripts }CVMActivation = swCVMDeportOnOffline = 1ClearClone = 1ScanDisks = 1DGOptions = { "-t -o usereplicatedev=only" })All 4 options(CVMDeportOnOffline, ClearClone, ScanDisks and DGOptions) are recommended with hardware-replicated disk groups.Patch ID: VRTSgms-8.0.0.3300* 4126370 (Tracking ID: 4129707)SYMPTOM:GMS rpm does not have changelogDESCRIPTION:Changelog in rpm will help to find missing incidents with respect to other version.RESOLUTION:Changelog is generated and added to GMS rpm.Patch ID: VRTSgms-8.0.0.2800* 4057427 (Tracking ID: 4057176)SYMPTOM:Rebooting the system results into emergency mode.DESCRIPTION:Module dependency files get corrupted due to parallel invocation of depmod.RESOLUTION:Serialized the invocation of depmod through file lock.Patch ID: VRTSgms-8.0.0.2300* 4108584 (Tracking ID: 4107753)SYMPTOM:The GMS module fails to load on RHEL8.7 minor kernel.DESCRIPTION:This issue occurs due to changes in the RHEL8.7 minor kernel.RESOLUTION:Modified existing modinst-gms script to accommodate the changes in the kernel and load the correct module.Patch ID: VRTSgms-8.0.0.2200* 4101000 (Tracking ID: 4100999)SYMPTOM:The GMS module fails to load on RHEL 8.7.DESCRIPTION:This issue occurs due to changes in the RHEL 8.7 kernel.RESOLUTION:GMS module is updated to accommodate the changes in the kernel and load as expected on RHEL 8.7.Patch ID: VRTSglm-8.0.0.3300* 4162713 (Tracking ID: 4126298)SYMPTOM:System may panic due to unable to handle kernel paging request and memory corruption could happen.DESCRIPTION:Panic may occur due to a race between a spurious wakeup and normal wakeup of thread waiting for glm lock grant. Due to the race, the spurious wakeup would have already freed a memory and then normal wakeup thread might be passing that freed and reused memory to wake_up function causing memory corruption and panic.RESOLUTION:Fixed the race between a spurious wakeup and normal wakeup threadsby making wake_up lock protected.Patch ID: VRTSglm-8.0.0.2300* 4108582 (Tracking ID: 4107754)SYMPTOM:The GLM module fails to load on RHEL8.7 minor kernel.DESCRIPTION:This issue occurs due to changes in the RHEL8.7 minor kernel.RESOLUTION:Modified existing modinst-glm script to accommodate the changes in the kernel and load the correct module.Patch ID: VRTSglm-8.0.0.2200* 4100995 (Tracking ID: 4100994)SYMPTOM:GLM module failed to load on RHEL8.7DESCRIPTION:The RHEL8.7 is new release and it has some changes in kernel which caused GLM module failed to loadon it.RESOLUTION:Added code to support GLM on RHEL8.7.Patch ID: VRTSodm-8.0.0.3300* 4162852 (Tracking ID: 4159291)SYMPTOM:The ODM module fails to load on RHEL-8.10.DESCRIPTION:This issue occurs due to changes in the RHEL-8.10 kernel.RESOLUTION:ODM module is updated to accommodate the changes in the kernel and load as expected on RHEL-8.10.* 4164549 (Tracking ID: 4129837)SYMPTOM:ODM rpm does not have changelogDESCRIPTION:Changelog in rpm will help to find missing incidents with respect to other version.RESOLUTION:Changelog is generated and added to ODM rpm.Patch ID: VRTSodm-8.0.0.3200* 4144128 (Tracking ID: 4126256)SYMPTOM:no symbol version warning for "ki_get_boot" in dmesg after SFCFSHA configurationDESCRIPTION:modpost is unable to read VEKI's Module.symvers while building ODM module, which results in no symbol version warning for "ki_get_boot" symbol of VEKI.RESOLUTION:Modified the code to make sure that modpost picks all the dependent symbols while building ODM module.* 4144301 (Tracking ID: 4118154)SYMPTOM:System may panic in simple_unlock_mem() when errcheckdetail enabled with stack trace as follows.simple_unlock_mem()odm_io_waitreq()odm_io_waitreqs()odm_request_wait()odm_io()odm_io_stat()vxodmioctl()DESCRIPTION:odm_io_waitreq() has taken a lock and waiting to complete the IO request but it is interrupted by odm_iodone() to perform IO and unlocked a lock taken by odm_io_waitreq(). So when odm_io_waitreq() tries to unlock the lock it leads to panic as lock was unlocked already.RESOLUTION:Code has been modified to resolve this issue.Patch ID: VRTSodm-8.0.0.3100* 4154894 (Tracking ID: 4144269)SYMPTOM:After installing, ODM fails to start.DESCRIPTION:Because of the VxFS version update, the ODM module needs to be repackaged due to aninternal dependency on the VxFS version.RESOLUTION:As part of this fix, the ODM module has been repackaged to support the updatedVxFS version.Patch ID: VRTSodm-8.0.0.2900* 4057432 (Tracking ID: 4056673)SYMPTOM:Rebooting the system results into emergency mode.DESCRIPTION:Module dependency files get corrupted due to parallel invocation of depmod.RESOLUTION:Serialized the invocation of depmod through file lock. Corrected vxgms dependency in odm service file.Patch ID: VRTSodm-8.0.0.2700* 4113912 (Tracking ID: 4113118)SYMPTOM:The ODM module fails to load on RHEL8.8.DESCRIPTION:This issue occurs due to changes in the RHEL8.8.RESOLUTION:Updated ODM to support RHEL 8.8.Patch ID: VRTSodm-8.0.0.2600* 4114656 (Tracking ID: 4114655)SYMPTOM:The ODM module fails to load on RHEL8.7 minor kernel 4.18.0-425.19.2.DESCRIPTION:This issue occurs due to changes in the RHEL8.7 minor kernel.RESOLUTION:Updated ODM to support RHEL 8.7 minor kernel 4.18.0-425.19.2.Patch ID: VRTSodm-8.0.0.2300* 4108585 (Tracking ID: 4107778)SYMPTOM:The ODM module fails to load on RHEL8.7 minor kernel.DESCRIPTION:This issue occurs due to changes in the RHEL8.7 minor kernel.RESOLUTION:Modified existing modinst-odm script to accommodate the changes in the kernel and load the correct module.Patch ID: VRTSodm-8.0.0.2200* 4100923 (Tracking ID: 4100922)SYMPTOM:ODM module failed to load on RHEL8.7DESCRIPTION:The RHEL8.7 is new release and it has some changes in kernel which caused ODM module failed to loadon it.RESOLUTION:Added code to support ODM on RHEL8.7.INSTALLATION PRE-REQUISITES:8.0 GASupported Platforms:RHEL8.6, RHEL8.8, RHEL8.10INSTALLING THE PATCH--------------------Run the Installer script to automatically install the patch:-----------------------------------------------------------Please be noted that the installation of this P-Patch will cause downtime.To install the patch perform the following steps on at least one node in the cluster:1. Copy the patch infoscale-rhel8_x86_64-Patch-8.0.0.3300.tar.gz to /tmp2. Untar infoscale-rhel8_x86_64-Patch-8.0.0.3300.tar.gz to /tmp/patch # mkdir /tmp/patch # cd /tmp/patch # gunzip /tmp/infoscale-rhel8_x86_64-Patch-8.0.0.3300.tar.gz # tar xf /tmp/infoscale-rhel8_x86_64-Patch-8.0.0.3300.tar3. Install the patch(Please be noted that the installation of this P-Patch will cause downtime.) # pwd /tmp/patch # ./installVRTSinfoscale800P3300 [<host1> <host2>...]You can also install this patch together with (8.0GA) base release using Install Bundles1. Download this patch and extract it to a directory2. Change to the Veritas InfoScale 8.0 directory and invoke the installer script with -patch_path option where -patch_path should point to the patch directory # ./installer -patch_path [<path to this patch>] [<host1> <host2>...]Install the patch manually:--------------------------Manual installation is not recommended.REMOVING THE PATCH------------------Manual uninstallation is not recommended.KNOWN ISSUES------------* Tracking ID: 4091160SYMPTOM: While doing two or more mount (of vxfs file system) operations in parallel, underneath an alreadyexisting vxfs mount point, if a force umount is attempted on the parent vxfs mount point, then sometimesthe force unmount operation hangs permanently.WORKAROUND: - There is no workaround, except rebooting the system.SPECIAL INSTRUCTIONS--------------------NONEOTHERS------NONE