libsas.rst (13804B)
1.. SPDX-License-Identifier: GPL-2.0 2 3========= 4SAS Layer 5========= 6 7The SAS Layer is a management infrastructure which manages 8SAS LLDDs. It sits between SCSI Core and SAS LLDDs. The 9layout is as follows: while SCSI Core is concerned with 10SAM/SPC issues, and a SAS LLDD+sequencer is concerned with 11phy/OOB/link management, the SAS layer is concerned with: 12 13 * SAS Phy/Port/HA event management (LLDD generates, 14 SAS Layer processes), 15 * SAS Port management (creation/destruction), 16 * SAS Domain discovery and revalidation, 17 * SAS Domain device management, 18 * SCSI Host registration/unregistration, 19 * Device registration with SCSI Core (SAS) or libata 20 (SATA), and 21 * Expander management and exporting expander control 22 to user space. 23 24A SAS LLDD is a PCI device driver. It is concerned with 25phy/OOB management, and vendor specific tasks and generates 26events to the SAS layer. 27 28The SAS Layer does most SAS tasks as outlined in the SAS 1.1 29spec. 30 31The sas_ha_struct describes the SAS LLDD to the SAS layer. 32Most of it is used by the SAS Layer but a few fields need to 33be initialized by the LLDDs. 34 35After initializing your hardware, from the probe() function 36you call sas_register_ha(). It will register your LLDD with 37the SCSI subsystem, creating a SCSI host and it will 38register your SAS driver with the sysfs SAS tree it creates. 39It will then return. Then you enable your phys to actually 40start OOB (at which point your driver will start calling the 41notify_* event callbacks). 42 43Structure descriptions 44====================== 45 46``struct sas_phy`` 47------------------ 48 49Normally this is statically embedded to your driver's 50phy structure:: 51 52 struct my_phy { 53 blah; 54 struct sas_phy sas_phy; 55 bleh; 56 }; 57 58And then all the phys are an array of my_phy in your HA 59struct (shown below). 60 61Then as you go along and initialize your phys you also 62initialize the sas_phy struct, along with your own 63phy structure. 64 65In general, the phys are managed by the LLDD and the ports 66are managed by the SAS layer. So the phys are initialized 67and updated by the LLDD and the ports are initialized and 68updated by the SAS layer. 69 70There is a scheme where the LLDD can RW certain fields, 71and the SAS layer can only read such ones, and vice versa. 72The idea is to avoid unnecessary locking. 73 74enabled 75 - must be set (0/1) 76 77id 78 - must be set [0,MAX_PHYS)] 79 80class, proto, type, role, oob_mode, linkrate 81 - must be set 82 83oob_mode 84 - you set this when OOB has finished and then notify 85 the SAS Layer. 86 87sas_addr 88 - this normally points to an array holding the sas 89 address of the phy, possibly somewhere in your my_phy 90 struct. 91 92attached_sas_addr 93 - set this when you (LLDD) receive an 94 IDENTIFY frame or a FIS frame, _before_ notifying the SAS 95 layer. The idea is that sometimes the LLDD may want to fake 96 or provide a different SAS address on that phy/port and this 97 allows it to do this. At best you should copy the sas 98 address from the IDENTIFY frame or maybe generate a SAS 99 address for SATA directly attached devices. The Discover 100 process may later change this. 101 102frame_rcvd 103 - this is where you copy the IDENTIFY/FIS frame 104 when you get it; you lock, copy, set frame_rcvd_size and 105 unlock the lock, and then call the event. It is a pointer 106 since there's no way to know your hw frame size _exactly_, 107 so you define the actual array in your phy struct and let 108 this pointer point to it. You copy the frame from your 109 DMAable memory to that area holding the lock. 110 111sas_prim 112 - this is where primitives go when they're 113 received. See sas.h. Grab the lock, set the primitive, 114 release the lock, notify. 115 116port 117 - this points to the sas_port if the phy belongs 118 to a port -- the LLDD only reads this. It points to the 119 sas_port this phy is part of. Set by the SAS Layer. 120 121ha 122 - may be set; the SAS layer sets it anyway. 123 124lldd_phy 125 - you should set this to point to your phy so you 126 can find your way around faster when the SAS layer calls one 127 of your callbacks and passes you a phy. If the sas_phy is 128 embedded you can also use container_of -- whatever you 129 prefer. 130 131 132``struct sas_port`` 133------------------- 134 135The LLDD doesn't set any fields of this struct -- it only 136reads them. They should be self explanatory. 137 138phy_mask is 32 bit, this should be enough for now, as I 139haven't heard of a HA having more than 8 phys. 140 141lldd_port 142 - I haven't found use for that -- maybe other 143 LLDD who wish to have internal port representation can make 144 use of this. 145 146``struct sas_ha_struct`` 147------------------------ 148 149It normally is statically declared in your own LLDD 150structure describing your adapter:: 151 152 struct my_sas_ha { 153 blah; 154 struct sas_ha_struct sas_ha; 155 struct my_phy phys[MAX_PHYS]; 156 struct sas_port sas_ports[MAX_PHYS]; /* (1) */ 157 bleh; 158 }; 159 160 (1) If your LLDD doesn't have its own port representation. 161 162What needs to be initialized (sample function given below). 163 164pcidev 165^^^^^^ 166 167sas_addr 168 - since the SAS layer doesn't want to mess with 169 memory allocation, etc, this points to statically 170 allocated array somewhere (say in your host adapter 171 structure) and holds the SAS address of the host 172 adapter as given by you or the manufacturer, etc. 173 174sas_port 175^^^^^^^^ 176 177sas_phy 178 - an array of pointers to structures. (see 179 note above on sas_addr). 180 These must be set. See more notes below. 181 182num_phys 183 - the number of phys present in the sas_phy array, 184 and the number of ports present in the sas_port 185 array. There can be a maximum num_phys ports (one per 186 port) so we drop the num_ports, and only use 187 num_phys. 188 189The event interface:: 190 191 /* LLDD calls these to notify the class of an event. */ 192 void sas_notify_port_event(struct sas_phy *, enum port_event, gfp_t); 193 void sas_notify_phy_event(struct sas_phy *, enum phy_event, gfp_t); 194 195The port notification:: 196 197 /* The class calls these to notify the LLDD of an event. */ 198 void (*lldd_port_formed)(struct sas_phy *); 199 void (*lldd_port_deformed)(struct sas_phy *); 200 201If the LLDD wants notification when a port has been formed 202or deformed it sets those to a function satisfying the type. 203 204A SAS LLDD should also implement at least one of the Task 205Management Functions (TMFs) described in SAM:: 206 207 /* Task Management Functions. Must be called from process context. */ 208 int (*lldd_abort_task)(struct sas_task *); 209 int (*lldd_abort_task_set)(struct domain_device *, u8 *lun); 210 int (*lldd_clear_task_set)(struct domain_device *, u8 *lun); 211 int (*lldd_I_T_nexus_reset)(struct domain_device *); 212 int (*lldd_lu_reset)(struct domain_device *, u8 *lun); 213 int (*lldd_query_task)(struct sas_task *); 214 215For more information please read SAM from T10.org. 216 217Port and Adapter management:: 218 219 /* Port and Adapter management */ 220 int (*lldd_clear_nexus_port)(struct sas_port *); 221 int (*lldd_clear_nexus_ha)(struct sas_ha_struct *); 222 223A SAS LLDD should implement at least one of those. 224 225Phy management:: 226 227 /* Phy management */ 228 int (*lldd_control_phy)(struct sas_phy *, enum phy_func); 229 230lldd_ha 231 - set this to point to your HA struct. You can also 232 use container_of if you embedded it as shown above. 233 234A sample initialization and registration function 235can look like this (called last thing from probe()) 236*but* before you enable the phys to do OOB:: 237 238 static int register_sas_ha(struct my_sas_ha *my_ha) 239 { 240 int i; 241 static struct sas_phy *sas_phys[MAX_PHYS]; 242 static struct sas_port *sas_ports[MAX_PHYS]; 243 244 my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0]; 245 246 for (i = 0; i < MAX_PHYS; i++) { 247 sas_phys[i] = &my_ha->phys[i].sas_phy; 248 sas_ports[i] = &my_ha->sas_ports[i]; 249 } 250 251 my_ha->sas_ha.sas_phy = sas_phys; 252 my_ha->sas_ha.sas_port = sas_ports; 253 my_ha->sas_ha.num_phys = MAX_PHYS; 254 255 my_ha->sas_ha.lldd_port_formed = my_port_formed; 256 257 my_ha->sas_ha.lldd_dev_found = my_dev_found; 258 my_ha->sas_ha.lldd_dev_gone = my_dev_gone; 259 260 my_ha->sas_ha.lldd_execute_task = my_execute_task; 261 262 my_ha->sas_ha.lldd_abort_task = my_abort_task; 263 my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set; 264 my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set; 265 my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2) 266 my_ha->sas_ha.lldd_lu_reset = my_lu_reset; 267 my_ha->sas_ha.lldd_query_task = my_query_task; 268 269 my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port; 270 my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha; 271 272 my_ha->sas_ha.lldd_control_phy = my_control_phy; 273 274 return sas_register_ha(&my_ha->sas_ha); 275 } 276 277(2) SAS 1.1 does not define I_T Nexus Reset TMF. 278 279Events 280====== 281 282Events are **the only way** a SAS LLDD notifies the SAS layer 283of anything. There is no other method or way a LLDD to tell 284the SAS layer of anything happening internally or in the SAS 285domain. 286 287Phy events:: 288 289 PHYE_LOSS_OF_SIGNAL, (C) 290 PHYE_OOB_DONE, 291 PHYE_OOB_ERROR, (C) 292 PHYE_SPINUP_HOLD. 293 294Port events, passed on a _phy_:: 295 296 PORTE_BYTES_DMAED, (M) 297 PORTE_BROADCAST_RCVD, (E) 298 PORTE_LINK_RESET_ERR, (C) 299 PORTE_TIMER_EVENT, (C) 300 PORTE_HARD_RESET. 301 302Host Adapter event: 303 HAE_RESET 304 305A SAS LLDD should be able to generate 306 307 - at least one event from group C (choice), 308 - events marked M (mandatory) are mandatory (only one), 309 - events marked E (expander) if it wants the SAS layer 310 to handle domain revalidation (only one such). 311 - Unmarked events are optional. 312 313Meaning: 314 315HAE_RESET 316 - when your HA got internal error and was reset. 317 318PORTE_BYTES_DMAED 319 - on receiving an IDENTIFY/FIS frame 320 321PORTE_BROADCAST_RCVD 322 - on receiving a primitive 323 324PORTE_LINK_RESET_ERR 325 - timer expired, loss of signal, loss of DWS, etc. [1]_ 326 327PORTE_TIMER_EVENT 328 - DWS reset timeout timer expired [1]_ 329 330PORTE_HARD_RESET 331 - Hard Reset primitive received. 332 333PHYE_LOSS_OF_SIGNAL 334 - the device is gone [1]_ 335 336PHYE_OOB_DONE 337 - OOB went fine and oob_mode is valid 338 339PHYE_OOB_ERROR 340 - Error while doing OOB, the device probably 341 got disconnected. [1]_ 342 343PHYE_SPINUP_HOLD 344 - SATA is present, COMWAKE not sent. 345 346.. [1] should set/clear the appropriate fields in the phy, 347 or alternatively call the inlined sas_phy_disconnected() 348 which is just a helper, from their tasklet. 349 350The Execute Command SCSI RPC:: 351 352 int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags); 353 354Used to queue a task to the SAS LLDD. @task is the task to be executed. 355@gfp_mask is the gfp_mask defining the context of the caller. 356 357This function should implement the Execute Command SCSI RPC, 358 359That is, when lldd_execute_task() is called, the command 360go out on the transport *immediately*. There is *no* 361queuing of any sort and at any level in a SAS LLDD. 362 363Returns: 364 365 * -SAS_QUEUE_FULL, -ENOMEM, nothing was queued; 366 * 0, the task(s) were queued. 367 368:: 369 370 struct sas_task { 371 dev -- the device this task is destined to 372 task_proto -- _one_ of enum sas_proto 373 scatter -- pointer to scatter gather list array 374 num_scatter -- number of elements in scatter 375 total_xfer_len -- total number of bytes expected to be transferred 376 data_dir -- PCI_DMA_... 377 task_done -- callback when the task has finished execution 378 }; 379 380Discovery 381========= 382 383The sysfs tree has the following purposes: 384 385 a) It shows you the physical layout of the SAS domain at 386 the current time, i.e. how the domain looks in the 387 physical world right now. 388 b) Shows some device parameters _at_discovery_time_. 389 390This is a link to the tree(1) program, very useful in 391viewing the SAS domain: 392ftp://mama.indstate.edu/linux/tree/ 393 394I expect user space applications to actually create a 395graphical interface of this. 396 397That is, the sysfs domain tree doesn't show or keep state if 398you e.g., change the meaning of the READY LED MEANING 399setting, but it does show you the current connection status 400of the domain device. 401 402Keeping internal device state changes is responsibility of 403upper layers (Command set drivers) and user space. 404 405When a device or devices are unplugged from the domain, this 406is reflected in the sysfs tree immediately, and the device(s) 407removed from the system. 408 409The structure domain_device describes any device in the SAS 410domain. It is completely managed by the SAS layer. A task 411points to a domain device, this is how the SAS LLDD knows 412where to send the task(s) to. A SAS LLDD only reads the 413contents of the domain_device structure, but it never creates 414or destroys one. 415 416Expander management from User Space 417=================================== 418 419In each expander directory in sysfs, there is a file called 420"smp_portal". It is a binary sysfs attribute file, which 421implements an SMP portal (Note: this is *NOT* an SMP port), 422to which user space applications can send SMP requests and 423receive SMP responses. 424 425Functionality is deceptively simple: 426 4271. Build the SMP frame you want to send. The format and layout 428 is described in the SAS spec. Leave the CRC field equal 0. 429 430open(2) 431 4322. Open the expander's SMP portal sysfs file in RW mode. 433 434write(2) 435 4363. Write the frame you built in 1. 437 438read(2) 439 4404. Read the amount of data you expect to receive for the frame you built. 441 If you receive different amount of data you expected to receive, 442 then there was some kind of error. 443 444close(2) 445 446All this process is shown in detail in the function do_smp_func() 447and its callers, in the file "expander_conf.c". 448 449The kernel functionality is implemented in the file 450"sas_expander.c". 451 452The program "expander_conf.c" implements this. It takes one 453argument, the sysfs file name of the SMP portal to the 454expander, and gives expander information, including routing 455tables. 456 457The SMP portal gives you complete control of the expander, 458so please be careful.