Hey! I have attached config.txt so you all can see what I'm doing. I added printing the error number as Marc suggested and the number appears to be -110 every time. [ 25.660006] CAN device driver interface [ 25.668720] spi_master spi0: will run message pump with realtime priority [ 25.676697] mcp251xfd spi0.1 can0: MCP2518FD rev0.0 (-RX_INT -MAB_NO_WARN +CRC_REG +CRC_RX +CRC_TX +ECC -HD c:40.00MHz m:20.00MHz r:17.00MHz e:16.66MHz) successfully initialized. [ 25.684900] mcp251xfd spi0.0 can1: MCP2518FD rev0.0 (-RX_INT -MAB_NO_WARN +CRC_REG +CRC_RX +CRC_TX +ECC -HD c:40.00MHz m:20.00MHz r:17.00MHz e:16.66MHz) successfully initialized. [ 28.098033] mcp251xfd spi0.1 rename4: renamed from can0 [ 28.175644] mcp251xfd spi0.0 can0: renamed from can1 [ 28.225891] mcp251xfd spi0.1 can1: renamed from rename4 [ 146.964971] mcp251xfd spi0.0: SPI transfer timed out [ 146.965023] spi_master spi0: failed to transfer one message from queue (ret=-110) [ 146.965216] mcp251xfd spi0.0 can0: CRC read error at address 0x0e0c (length=4, data=00 00 00 00, CRC=0x0000) retrying. [ 146.965247] mcp251xfd spi0.0 can0: CRC read error at address 0x0e0c (length=4, data=00 00 00 00, CRC=0x0000) retrying. [ 146.965277] mcp251xfd spi0.0 can0: CRC read error at address 0x0e0c (length=4, data=00 00 00 00, CRC=0x0000) retrying. [ 146.965286] mcp251xfd spi0.0 can0: CRC read error at address 0x0e0c (length=4, data=00 00 00 00, CRC=0x0000). [ 146.965331] mcp251xfd spi0.0 can0: CRC read error at address 0x0000 (length=4, data=00 00 00 00, CRC=0x0000) retrying. [ 146.965360] mcp251xfd spi0.0 can0: CRC read error at address 0x0000 (length=4, data=00 00 00 00, CRC=0x0000) retrying. [ 146.965389] mcp251xfd spi0.0 can0: CRC read error at address 0x0000 (length=4, data=00 00 00 00, CRC=0x0000) retrying. [ 146.965397] mcp251xfd spi0.0 can0: CRC read error at address 0x0000 (length=4, data=00 00 00 00, CRC=0x0000). [ 146.965413] A link change request failed with some changes committed already. Interface can0 may have been left with an inconsistent configuration, please check. Regarding the discussion about Kconfig flags, I went ahead and rebuilt kernel 5.10.44 using a config that was essentially arch/arm/configs/bcm2711_defconfig with these additions needed to get our I2S working. This should have undone the switch to ONDEMAND governor and enabling 1000 Hz clock. 1030a1031 > CONFIG_SND_RPI_I2S_AUDIO_WM8782=m 1040a1042 > CONFIG_SND_SOC_WM8782=m My RPi and HAT have worked very reliably with the older buster image and customized (same tweaks as mentioned in last email) kernel 4.19.73, in that kernel I'm using MCP25XXFD driver from msperl which under 5.10.Y kernel is having issues too. I only upgraded everything on my system at the end of last week, so hardware has been OK very recently. Keep in mind I'm not seeing a total failure, I do occasionally see everything work correctly and I can run the ip link setup command without issue, it's just not common and it seems fully removing power from the system and reapplying seems to help, but not every time, so maybe it's a coincidence. It could be an issue of subsequent configurations of the controller after the initial setup on power application, but I'd expect it work after every power yank I think. I wouldn't feel comfortable reverting my /boot/config.txt to a stock one and a default setup of the 40-pin header, at least not with my HAT attached which includes the CAN controllers AND circuitry to supply power to RPi from a 12V rail. Thanks, Josh Q -----Original Message----- From: Patrick Menschel <menschel.p@xxxxxxxxx> Sent: Wednesday, June 23, 2021 1:24 AM To: Joshua Quesenberry <engnfrc@xxxxxxxxx>; Marc Kleine-Budde <mkl@xxxxxxxxxxxxxx> Cc: kernel@xxxxxxxxxxxxxx; linux-can@xxxxxxxxxxxxxxx Subject: Re: MCP2518FD Drivers Rarely Working with Custom Kernel 5.10.Y Am 23.06.21 um 04:59 schrieb Joshua Quesenberry: > Thank you Marc, I had tried finding a Linux CAN forum, but > unfortunately searching for "CAN" in Google is about the most > unhelpful search term one could use... so thanks for replying and > getting me to a more appropriate audience. > > Reverting my system back to where CAN was working will probably be > challenging. Our main goal was to get Boot from USB on the RPi > enabled, but this unfortunately meant upgrading every piece of > software and firmware available... previously we were still on Buster, > but the OS snapshot was from Spring 2020 (if not Fall/Winter 2019), if > not earlier, the firmware was much older, and the kernel was 4.19.73, > wherein the MCP251XFD driver didn't exist yet. So getting back there > will mean throwing a saved SD Card image on from Spring 2020 and then > trying to figure out how to force downgrade the firmware. A colleague > started this upgrade process for another project and was seeing these > same results on two separate RPi, he did the OS and firmware upgrades, > but I did the building of the 5.10.17 kernel. So including those two > RPi and mine, that's three total systems with mostly non-working CAN > where it had been working fine, my system has slightly newer RPi > firmware now and the 5.10.44 kernel, the hope was maybe I'd pick up a > patch somewhere, but no such luck. If you still think it would be > beneficial to go through the effort of downgrading everything to > verify the hardware I can do that, but just want to make sure before I > start that since it'll take a while. > > I updated spi.c to include printing the error number as you requested > and that's all baking now. When I get into work in the morning (US > EST) I'll get the changes deployed and try it out. Since this issue is > a very high failure rate, getting a log shouldn't be an issue. > > Some background on the custom kernel... when I switched to the 5.10.Y > branch, I used arch/arm/configs/bcm2711_defconfig as my base config > and then switched on preempt, switched to 1000Hz kernel timer, > switched the default governor from powersave to ondemand, switched on > debug flag (CONFIG_DEBUG_USER=y), enabled a few different CAN drivers > we may encounter, and enabled some stuff for the WM8782 I2S chip. I > probably should have recreated my config after 5.10.44, but I hadn't > considered till this writing, looking at this diff there a few bits > that are new I probably could benefit from including, but I don't see > anything that I'd be concerned about. > > `diff bcm2711_defconfig hel_bcm2711_lowlatency_defconfig` > 15d14 > < CONFIG_ATA=m > 43d41 > < CONFIG_BH1750=m > 53c51 > < CONFIG_BLK_DEV_NVME=y > --- >> CONFIG_BLK_DEV_NVME=m > 120c118 > < CONFIG_CAN_J1939=m > --- >> CONFIG_CAN_KVASER_USB=m > 123a122,123 >> CONFIG_CAN_MCP25XXFD=m >> CONFIG_CAN_PEAK_USB=m > 127d126 > < CONFIG_CCS811=m > 155c154 > < CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE=y > --- >> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y > 158,159c157 > < CONFIG_CPU_FREQ_GOV_ONDEMAND=y > < CONFIG_CPU_FREQ_GOV_PERFORMANCE=y > --- >> CONFIG_CPU_FREQ_GOV_POWERSAVE=y > 184a183 >> CONFIG_DEBUG_USER=y > 209d207 > < CONFIG_DRM_PANEL_JDI_LT070ME05000=m > 319a318 >> CONFIG_GENERIC_PHY=y > 325d323 > < CONFIG_GPIO_PCA953X_IRQ=y > 395a394 >> CONFIG_HZ_1000=y > 561d559 > < CONFIG_IR_TOY=m > 826d823 > < CONFIG_NF_LOG_ARP=m > 828d824 > < CONFIG_NF_LOG_NETDEV=m > 950c946 > < CONFIG_PREEMPT_VOLUNTARY=y > --- >> CONFIG_PREEMPT=y > 957d952 > < CONFIG_QCA7000_UART=m > 994d988 > < CONFIG_RPI_POE_POWER=m > 1040a1035 >> # CONFIG_RTC_HCTOSYS is not set > 1044,1045d1038 > < CONFIG_SATA_AHCI=m > < CONFIG_SATA_MV=m > 1054d1046 > < CONFIG_SENSIRION_SGP30=m > 1134a1127 >> CONFIG_SND_RPI_I2S_AUDIO_WM8782=m > 1149a1143 >> CONFIG_SND_SOC_WM8782=m > > The /boot/config.txt I included in the forum posts mentioned is > tweaking the 40-pin header quite a bit from the default setup, we're > using many of the pins for our HAT and planned for possibly adding > more in the future. Hi, it would help to find a reference to that config.txt . Regarding the changed Kconfig flags, I would suspect everything that owns a =y to be the culprit, especially everything that has connections to a clock. Ever since the first rpi3, clocks are unreliable in general due to the frequency governor. The rpi guys did there best to get rid of most of the initial problems but the root cause remains. The interesting question is, does a stock raspbian buster work with your hardware and that config.txt? I'm running a stock raspbian buster on a rpi3b+ with seeed can fd hat v2 24/7 for a couple of month now and did not expierence any problems. Regards, Patrick
# For more options and information see # http://rpf.io/configtxt # Some settings may impact device functionality. See link above for details dtdebug=1 disable_splash=1 boot_delay=0 # uncomment if you get no picture on HDMI for a default "safe" mode #hdmi_safe=1 # uncomment this if your display has a black border of unused pixels visible # and your display can output without overscan #disable_overscan=1 # uncomment the following to adjust overscan. Use positive numbers if console # goes off screen, and negative if there is too much border #overscan_left=16 #overscan_right=16 #overscan_top=16 #overscan_bottom=16 # uncomment to force a console size. By default it will be display's size minus # overscan. #framebuffer_width=1280 #framebuffer_height=720 # uncomment if hdmi display is not detected and composite is being output #hdmi_force_hotplug=1 hdmi_force_mode=1 # uncomment to force a specific HDMI mode (this will force VGA) #hdmi_group=1 #hdmi_mode=1 hdmi_group=1 hdmi_mode=3 # 480p 60Hz H #hdmi_mode=9 # 240p 60Hz H # uncomment to force a HDMI mode rather than DVI. This can make audio work in # DMT (computer monitor) modes #hdmi_drive=2 # uncomment to increase signal to HDMI, if you have interference, blanking, or # no display #config_hdmi_boost=4 # uncomment for composite PAL #sdtv_mode=2 #uncomment to overclock the arm. 700 MHz is the default. #arm_freq=800 #arm_freq=1500 #gpu_freq=600 #core_freq=600 #h264_freq=600 #isp_freq=600 #v3d_freq=600 # Uncomment some or all of these to enable the optional hardware interfaces dtparam=i2c_arm=on dtparam=i2s=on dtparam=spi=off enable_uart=0 # Uncomment this to enable the lirc-rpi module #dtoverlay=lirc-rpi # Additional overlays and parameters are documented /boot/overlays/README # Enable audio (loads snd_bcm2835) dtparam=audio=on [pi4] # Enable DRM VC4 V3D driver on top of the dispmanx display stack dtoverlay=vc4-fkms-v3d max_framebuffers=2 [all] #dtoverlay=vc4-fkms-v3d start_x=1 gpu_mem=512 # GPS dtoverlay=uart3 # Reserved for HAT IDs dtoverlay=i2c0 dtparam=pins_0_1=on dtparam=combine=off # Power Supervisor, IMU, RPi I2C Bus dtoverlay=i2c1 # CAN 0/1 dtoverlay=spi0-cs dtparam=cs0_pin=8 dtparam=cs1_pin=7 # Reserved dtoverlay=spi5-2cs dtparam=cs0_pin=12 dtparam=cs0_spidev=disabled dtparam=cs1_pin=26 dtparam=cs1_spidev=disabled # # CAN 0 # dtoverlay=mcp2517fd-spi0_0-can0 # dtparam=oscillator=40000000 # dtparam=spimaxfrequency=20000000 # dtparam=interrupt=24 # # # CAN 1 # dtoverlay=mcp2517fd-spi0_1-can1 # dtparam=oscillator=40000000 # dtparam=spimaxfrequency=20000000 # dtparam=interrupt=25 # CAN 0 dtoverlay=mcp251xfd,spi0-0,interrupt=24,oscillator=40000000,speed=20000000 # CAN 1 dtoverlay=mcp251xfd,spi0-1,interrupt=25,oscillator=40000000,speed=20000000 dtoverlay=hel-wm8782,alsaname=mic
